Micro-archiving the Bluesky Jetstream firehose with Python and sqlite-utils

I was reading the Hacker News discussion about Bluesky's new Jetstream architecture this week. The pervasive assumption running through the comments is that consuming a global social media firehose inherently requires an intimidating, enterprise-grade infrastructure. The word firehose itself has acquired a lot of expensive baggage. Over the last decade, we've been trained to associate it with distributed load balancers, dedicated Redis caching layers, and massive monthly cloud provider bills. It’s a highly understandable architectural trauma. It's born directly from years of dealing with opaque vendor silos—most notably the agonizing death of the Twitter API—that deliberately gatekept their network traffic behind complex engineering requirements and exorbitant enterprise tiers.

Bluesky recently published a post introducing Jetstream v2, and reading through the accompanying protocol documentation reveals how drastically they’ve bypassed that traditional enterprise overhead. To understand why this works, we need to define a strict technical boundary between the heavy demands of a traditional federated sync protocol and the much dumber, drastically simplified pipeline this new system actually delivers.

The core AT Protocol synchronization firehose relies on dense, heavily structured CBOR and CAR files. These are designed to cryptographically verify the entire network state across federated instances. That level of rigor is undeniably heavy—it requires significant computational overhead just to keep up with the sheer global event volume, unpack the cryptographic signatures, and maintain state. This architecture targets massive relay operators rather than individual hobbyists.

The Jetstream alternative entirely abandons that cryptographic verification overhead. Crucially: if there is no need for cryptographic proof of every single network action, you can drop the heavy lifting entirely. A global data stream is aggressively reduced to an unfiltered torrent of standard JSON payloads, flowing constantly over an unauthenticated WebSocket connection.

Listeners merely open the pipe and accept the text, completely sidestepping the API keys, rate-limiting headers, and complex OAuth handshakes that usually gatekeep this kind of data. You can test this right now with a single terminal command against the literal wss://jetstream.us-east.bsky.network unauthenticated WebSocket endpoint:

websocat wss://jetstream.us-east.bsky.network/subscribe

The connection immediately yields a cascade of objects formatted like this:

{
  "did": "did:plc:ragtjsm2j2vjnwkdzbg7nlc3",
  "time_us": 1729094073000000,
  "kind": "commit",
  "commit": {
    "rev": "3kv2z...",
    "operation": "create",
    "collection": "app.bsky.feed.post",
    "rkey": "3kv2z...",
    "record": {
      "$type": "app.bsky.feed.post",
      "createdAt": "2024-10-16T15:54:33.000Z",
      "text": "Just setting up my bsky"
    }
  }
}

Observers parsing this pipe receive basic, readable key-value pairs—a top-level commit object containing a nested record dictionary, which ultimately holds the raw text of the user's post in a predictable app.bsky.feed.post namespace. It feels exactly like the golden age of the web: just plain text moving over an open socket.

Watching a terminal window scroll indefinitely has limited utility, but grabbing those incoming strings and piping them into a local database fundamentally changes the stakes of network participation. I was reading a thread recently where a developer demonstrated doing exactly this, using the .insert() method from the sqlite-utils Python library to effortlessly map inbound text to a persistently updated local SQLite database. By turning on alter=True, you can do this without writing any tedious schema migrations.

A complete, functioning implementation of that concept requires barely a screen of logic:

import json
import asyncio
import websockets
from sqlite_utils import Database

async def stream_firehose():
    db = Database("bluesky_archive.db")
    uri = "wss://jetstream.us-east.bsky.network/subscribe?wantedCollections=app.bsky.feed.post"
    
    async with websockets.connect(uri) as ws:
        while True:
            message = await ws.recv()
            record = json.loads(message)
            
            if record.get("kind") == "commit":
                data = record.get("commit", {}).get("record", {})
                if data:
                    db["posts"].insert(
                        {
                            "text": data.get("text"),
                            "createdAt": data.get("createdAt"),
                            "did": record.get("did")
                        },
                        alter=True
                    )

asyncio.run(stream_firehose())

This script connects to the public endpoint, isolates the new text posts using a basic query parameter collection filter, and streams those parsed dictionaries directly into a local .db file in real time. Because the sqlite-utils tooling automatically alters the database table schema to accommodate any unexpected new keys in the incoming JSON payload, the incoming data dynamically generates the SQLite storage container on arrival. Bypassing the usual ORM boilerplate makes this the ultimate lazy hacker trick.

Once it's running, querying your local archive is just a matter of running standard SQL against it. You can instantly pull the latest posts using a quick terminal command:

sqlite-utils rows bluesky_archive.db posts --limit 3

This instantly proves the value of the exercise: you now have a permanent, queryable local index of a global network stream, completely bypassing the vendor's API!

Because the payload is stripped of heavy protocol overhead, an explicitly hypothetical localized setup like this could run quietly in the background of a standard laptop or a cheap Raspberry Pi for weeks without spiking CPU temperatures. Calculating the raw storage requirements quickly demonstrates the sheer affordability of this pipeline. If a stripped-down post record consumes around 300 bytes, filtering the stream for a niche topic or a specific community list might yield only 3,000 posts a day. That's just under a megabyte of data daily. The resulting local archive grows by mere hundreds of megabytes a year, costing effectively nothing to host.

This architectural approach unlocks a practice I've started calling micro-archiving. We have conditioned ourselves to accept proprietary data centers as the only viable home for social data, largely because managing it at scale is assumed to be prohibitively expensive. The AT Protocol—particularly when exposed via these lightweight, unauthenticated Jetstream pipes—aggressively pushes the burden of data retention away from highly capitalized platforms. It allows individuals to trivially build local, personal data warehouses populated exclusively by the exact semantic information they find valuable.

If the future of the web relies on trusting VC-funded platforms to retain our history, we’re in a whole heap of trouble. Micro-archiving gives us a completely different path: pulling exactly the data we care about, saving it to a file on our own disk, and building our own weird tools on top of it. This stuff is so much fun.