Executing a SQLite database (and why strace failed me!)

Yesterday I was reading a wild Hacker News discussion about Farid Zakaria's brilliant project where a SQLite database executes itself.

Someone in the thread asked how this was even physically possible, and my immediate reaction was pure confusion. My mental model has always been that a file you run is either a compiled ELF binary (like a Go or Rust program) or a text file that starts with a shebang (like #!/bin/bash). The idea of typing ./my_database.db into the terminal and having it physically execute queries against itself feels deeply cursed. Databases are meant to be queried by other programs, not run directly as standalone executables! (How does the kernel even know what to do with it??)

To understand exactly how the operating system handles executing files, the natural instinct is to look at strace output. If someone creates a tiny, dummy SQLite database, makes it executable, and runs strace on it, the kernel will reject it out of the box. The sequence looks exactly like this:

$ sqlite3 my_database.db "CREATE TABLE users (id INTEGER);"
$ chmod +x my_database.db
$ strace ./my_database.db

Here is the raw output that comes back:

execve("./my_database.db", ["./my_database.db"], 0x7ffd59b12a80 /* 53 vars */) = -1 ENOEXEC (Exec format error)
strace: exec: Exec format error

This is SO cool. We get to see the exact system call responsible for starting programs: execve. Usually, execve takes a path to a file, sets up the memory, and starts running the instructions. But here, the kernel was asked to execute a database file, and the kernel immediately returned -1 ENOEXEC, which translates to "Exec format error". Basically, the kernel looked at the file, threw its hands in the air, and said "I have absolutely no idea what you want me to do with this."

But how does it know? Linux famously does not care about file extensions like .db or .txt. It turns out, when execve is called, the kernel physically reads the first few bytes of the file to look for a "magic number."

This can be seen in action by dropping down to the terminal and using hexdump -C to look at the first 16 bytes of a standard system binary vs the dummy database.

$ hexdump -C /bin/ls | head -n 1
00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|

$ hexdump -C my_database.db | head -n 1
00000000  53 51 4c 69 74 65 20 66  6f 72 6d 61 74 20 33 00  |SQLite format 3.|

The execution process is basically just a giant matching game! When the kernel sees 7f 45 4c 46 (which translates to .ELF), it has a hardcoded rule that says "Ah, pass this to the ELF loader." (Similarly, if it sees 23 21—which is #!—it knows to read the rest of the line to find the script's interpreter). But when it looks at the database, it reads SQLite format 3. Since SQLite isn't in the kernel's hardcoded routing table, it panics and throws the ENOEXEC error.

So how is the ENOEXEC error fixed to make Farid's self-executing database trick work?

Enter an incredible kernel feature called binfmt_misc (which stands for "binary format miscellaneous"). It allows dynamic additions of new magic numbers to the kernel's internal routing table without recompiling anything. (This is actually exactly how systems are configured to automatically launch Windows .exe files via Wine, or Java .class files!).

A special filesystem just needs to be mounted, and a weirdly formatted string written into it. To make the database executable, a rule that looks roughly like this would be written:

:sqlite:M::SQLite format 3::/usr/bin/sqlite3:

Here is what that chaotic string is actually telling the kernel:

• sqlite: "Name this rule 'sqlite'."
• M: "We are matching on a Magic number (instead of a file extension)."
• SQLite format 3: "If the first bytes of the file match this literal string..."
• /usr/bin/sqlite3: "...don't throw an error! Just silently pass the file to this interpreter."

Once this is registered, calling execve("./my_database.db") works perfectly.

But here is the thing that can be totally confusing when trying to trace a custom executable like this: the internal routing trick the kernel performs is completely invisible to user-space tools! If a file supported by binfmt_misc is run through strace, you might expect to see the kernel elegantly handing off the database to the custom handler. You might think there would be some extra syscalls showing the lookup process. Instead, the strace output just shows a successful execve returning 0, and then suddenly the new program is running. Once the execve request physically enters kernel space, the magic-number swapping happens in a total black box.

(It's a great reminder that strace only shows the boundary between user space and kernel space, not what the kernel is doing internally!)

Thinking about this whole process made me realize that executing a program isn't fundamentally magic at all. It blows my mind that deep down, the kernel is essentially just running a massive switch statement on the first four bytes of a file. And even better, if you know the right incantations to talk to the kernel, you can just hot-patch that switch statement dynamically. It is honestly incredible that modern operating systems let you hotwire their core routing logic with a single chaotic string of text!