Thanks for the design discussion 😅
> vacuum
You're right. I suspect this issue will arise because blobs take most of the space, so here's an idea: create separate databases for blobs, one for each first byte, so that they can all be vacuumed safely. Your 6 TB database becomes 1 database for meta stuff (I suspect less than 10 GB) and 256 databases for blobs, each statistically around 6*1024/256 = 24 GB. Way more tractable. Deletions happen in the meta db, blobs dbs lag behing and canbe garbage collected asynchronously.
> blob interface, base64
My point is that stringy key-values are stored directly in the database, and blobs are blobs whose content are irrelevant to processing. You shouldn't need to read a blob, decipher it, read the first line-oh but actually it's crlf not cr and all that crap to function. sqlite provides the structure. blobs contain files content and nothing else.
> configurable batching of chunk transactions
sqlite has that
rakoo
npub1y4…jh9df
2026-10-07 01:28:04 UTC
in reply to nevent1q…uucw
Author Public Key
npub1y4k8zwggprw5xkyck08lz9ng82ujf43c0ml22grvf56mray42xmq8jh9dfPublished at
2026-10-07 01:28:04 UTCEvent JSON
{
"id": "f2cb0cba0a3078106345061ccc7b7d8c3d027d5cb699fbc9ef988b1ea3e53c21",
"pubkey": "256c71390808dd435898b3cff116683ab924d6387efea5206c4d35b1f49551b6",
"created_at": 1791336484,
"kind": 1,
"tags": [
[
"p",
"9096c95c05b71521d6812b521bc02db89f6677a33ce3670d9af2bcfee5042497"
],
[
"e",
"3acaeda9aba45f18c5f17442b5187574dcf07ad2bb0c7810caf2f9aeee471550",
"",
"root",
"8ab44a84fb757fea3c983e68a9bcc8e331fa49ed99cfcfd06a4499d542eae93c"
],
[
"p",
"8ab44a84fb757fea3c983e68a9bcc8e331fa49ed99cfcfd06a4499d542eae93c"
],
[
"e",
"f6f1d8ae2498b5ed41d614797c15cca0e14b82cd791b26a589707c093335c4a3",
"",
"reply",
"8ab44a84fb757fea3c983e68a9bcc8e331fa49ed99cfcfd06a4499d542eae93c"
],
[
"p",
"256c71390808dd435898b3cff116683ab924d6387efea5206c4d35b1f49551b6"
],
[
"proxy",
"https://blah.rako.space/objects/571cfa93-fb87-43dd-acba-3b165dda03c4",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://blah.rako.space/objects/571cfa93-fb87-43dd-acba-3b165dda03c4",
"pink.momostr"
],
[
"-"
]
],
"content": "Thanks for the design discussion 😅\r\n\r\n\u003e vacuum\r\n\r\nYou're right. I suspect this issue will arise because blobs take most of the space, so here's an idea: create separate databases for blobs, one for each first byte, so that they can all be vacuumed safely. Your 6 TB database becomes 1 database for meta stuff (I suspect less than 10 GB) and 256 databases for blobs, each statistically around 6*1024/256 = 24 GB. Way more tractable. Deletions happen in the meta db, blobs dbs lag behing and canbe garbage collected asynchronously.\r\n\r\n\u003e blob interface, base64\r\n\r\nMy point is that stringy key-values are stored directly in the database, and blobs are blobs whose content are irrelevant to processing. You shouldn't need to read a blob, decipher it, read the first line-oh but actually it's crlf not cr and all that crap to function. sqlite provides the structure. blobs contain files content and nothing else.\r\n\r\n\u003e configurable batching of chunk transactions\r\n\r\nsqlite has that",
"sig": "4f37bd727aa3b6f0ace50e6e1b6a80c6579e3997178d5150d5409dfc32cc87178922068e7be0378929655fbcf392f4b85fb7fb3b55a44226cfdcb1e0f5684fb8"
}