IPng Networks
IPngNetworks@ublog.tech
I found some non-hyperscaler disks! 26x18TB will become our next RustFS (minio replacement) S3 tanks (2x12x18TB per node). Wish me luck!
1
0
0
Beware, RustFS is buggy slop that does not understand the basics of distributed storage or even not hardcoding root credentials.
1
1
0
✧✦Catherine✦✧
whitequark@treehouse.systems@q3k @IPngNetworks so what's actually a reasonable minio replacement these days? garage isn't it, rustfs isn't it, others?
5
0
0
@whitequark @q3k @IPngNetworks i haven't yet found one :/
0
0
0
@whitequark @q3k @IPngNetworks what’s wrong with garage? speaking from a perspective of someone who “doesn’t really get” s3, it felt as promising as ceph for making something distributed
2
0
0
✧✦Catherine✦✧
whitequark@treehouse.systems@domi @q3k @IPngNetworks too immature for running in production (i went with a commercial service instead)
1
0
0
@domi @whitequark @IPngNetworks I don't trust it, but given RustFS seems to be the alternative then I'm now fine with it. Anything but hype-driven slop, please.
0
0
0
@whitequark @q3k @IPngNetworks https://github.com/seaweedfs/seaweedfs maybe? but I basically only know it exists
1
0
0
✧✦Catherine✦✧
whitequark@treehouse.systems
0
0
0
Krzysztof Sakrejda
wronglang@bayes.club@whitequark @domi @q3k @IPngNetworks I'm so glad this is the extent of the criticism, these days it's basically an endorsement
1
0
0
✧✦Catherine✦✧
whitequark@treehouse.systems@wronglang @domi @q3k @IPngNetworks it is true that the bar is very low but as someone who is ultimately responsible for data stored by others that's still a problem for me
0
0
0
@whitequark @q3k @IPngNetworks ceph via rgw I guess, not the lightest of things, but then again, distributed, reliable storage is not a light thing. I do wonder, why does minio need replacing? haven't checked it in a while.
1
0
0
✧✦Catherine✦✧
whitequark@treehouse.systems@jkmcnk @q3k @IPngNetworks they went full-on AI-only and also it's no longer open source
0
0
0
Jeroen Massar
jeroen@secluded.ch@whitequark @q3k @IPngNetworks What is off with Garage (https://garagehq.deuxfleurs.fr) ? On surface looks quite nice, as are others of their projects.
is it really that in 2026 there is no ‘good functioning' solution for this?
1
0
0
✧✦Catherine✦✧
whitequark@treehouse.systems@jeroen @q3k @IPngNetworks i don't have a lot of confidence in their GC implementation https://garagehq.deuxfleurs.fr/documentation/design/internals/
1
0
0
Charlotte 
/Cinny 
charlotte
@whitequark @jeroen @q3k @IPngNetworks we’re running garage and the main issue we got is their metadata db
LMDB loves corrupting itself if not properly closed/shut down, sqlite is significantly slower, fjall (experimental) causes the block error rate to just go up all the time (even though all the data accesses fine?)
3
0
2
✧✦Catherine✦✧
whitequark@treehouse.systems@charlotte @jeroen @q3k @IPngNetworks yeah this is the type of issue i was concerned about
not a total deal breaker and will probably be fixed in the long run, but a problem for using it right now
0
0
0
Maciej Barć
xgqt@functional.cafe@charlotte @jeroen @q3k @whitequark @IPngNetworks
I mean I'd rather take slowness of SQLite over corruption.
Ceph is insane to set up and operate.
1
0
0
Charlotte /Cinny
charlotte
@xgqt @jeroen @q3k @whitequark @IPngNetworks i might set up a ceph cluster if i could afford a ram upgrade for the machine
0
0
0
Jeroen Massar
jeroen@secluded.ch@charlotte @whitequark @q3k @IPngNetworks
@hyc any commentary on the above? ;)
1
0
0
Howard Chu @ Symas
hyc@mastodon.social@jeroen @charlotte @whitequark @q3k @IPngNetworks with its default settings, fsync on every txn commit, LMDB's crash consistency is absolutely perfect. If you turn off fsync to gain write throughput, then guarantees are weaker. But an application-level crash can never corrupt it. An OS or hardware- level crash can, when fsyncs are omitted. But if the underlying filesystem does ordered writes, then even OS/hardware crashes are no problem.
2
0
0
Howard Chu @ Symas
hyc@mastodon.social@jeroen @charlotte @whitequark @q3k @IPngNetworks I've never looked at their code. If Garage uses LMDB with unsafe settings, that's on them.
0
0
1
Jeroen Massar
jeroen@secluded.ch@hyc @charlotte @whitequark @q3k @IPngNetworks That is what I expected for your code ;)
They even call it out:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-v2/doc/book/reference-manual/known-issues.md#lmdb-metadata-corruption
The code seems to have a fsync flag:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-v2/src/db/lmdb_adapter.rs#L43
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-v2/src/db/open.rs
and guess what:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-v2/doc/book/reference-manual/configuration.md#metadata_fsync oh my... so Garage is intended for cheap hardware, but they turn off metadata_fsync per default and thus MDB_NOSYNC. There is no option to toggle MDB_NOMETASYNC.
Well, I think we found the culprit…. the need for speed over consistency.
1
0
0
IPng Networks
IPngNetworks@ublog.tech@jeroen @hyc @charlotte @whitequark @q3k
what an absolutely lovely thread this has become :) I think I may reconsider RustFS and perhaps start with SeaweedFS, which I have used before as file storage for Pixelfed and Loops.
I am also surprised that the amount of blobstore choices is fairly limited in the FOSS world.
Thanks for the operational notes, all!
0
0
0