Every like, flag and label on ZenMemes is its own small text file, plus a .asc file when it's signed. That's a good format: one event per file, easy to read, easy to verify, easy to retract. The cost is file count. Bytes aren't the problem (a signed like is around 600 bytes, and git packs near-identical files well). The problem is the working tree filling up with many thousands of tiny files, and every full read-model rebuild having to open each one.
Changing the record format to fix this, say by letting one record tag many targets, would make retraction messy and blur the "one author, one target, one tag is one vote" rule. I'd rather fix it once, below the format, for every record family at the same time.
The idea: periodically pack old records into archive files. Say once a month, records older than some cutoff move out of records/post-reactions/ and the like into one bundle per family per month. A bundle holds each record's exact original bytes and its signature, with an index of record IDs and offsets. Nothing about a record changes. It's the same bytes, with the same signature, verified the same way. Only where it lives changes.
For this to work:
Signatures cover exact bytes, so packing must never normalize anything. Unpacking a bundle should give back byte-identical files.
Readers need one lookup path that checks loose files first, then bundles, so the rest of the code doesn't care where a record sits.
Retraction and history purge have to reach into bundles. The simplest rule is probably to rewrite the affected bundle without the purged record, since purges are rare.
New records stay loose. Packing only touches old, settled ones.
Before building any of it, it's worth measuring: how many record files prod has per family today, and how much of a full rebuild's time goes to opening files rather than processing them. If the file count is still small, this can wait. If rebuild time is mostly file I/O, packing might also help with the slow rebuilds we've been seeing.
#meta #storage
#
Changing the record format to fix this, say by letting one record tag many targets, would make retraction messy and blur the "one author, one target, one tag is one vote" rule. I'd rather fix it once, below the format, for every record family at the same time.
The idea: periodically pack old records into archive files. Say once a month, records older than some cutoff move out of records/post-reactions/ and the like into one bundle per family per month. A bundle holds each record's exact original bytes and its signature, with an index of record IDs and offsets. Nothing about a record changes. It's the same bytes, with the same signature, verified the same way. Only where it lives changes.
For this to work:
Signatures cover exact bytes, so packing must never normalize anything. Unpacking a bundle should give back byte-identical files.
Readers need one lookup path that checks loose files first, then bundles, so the rest of the code doesn't care where a record sits.
Retraction and history purge have to reach into bundles. The simplest rule is probably to rewrite the affected bundle without the purged record, since purges are rare.
New records stay loose. Packing only touches old, settled ones.
Before building any of it, it's worth measuring: how many record files prod has per family today, and how much of a full rebuild's time goes to opening files rather than processing them. If the file count is still small, this can wait. If rebuild time is mostly file I/O, packing might also help with the slow rebuilds we've been seeing.
#meta #storage