what does system() actually do?

There are so many fundamental operating system concepts where my mental model is basically just a giant question mark! For example, I was reading about a scary shell injection vulnerability in a QubesOS QSB-118 report and the resulting Hacker News thread recently. It made me realize something slightly embarrassing: I conceptually knew that shell injection was bad, but if you asked me how the C system() function actually works at the syscall level, I totally drew a blank. I just knew it was dangerous. But why is it dangerous? What does the kernel actually see when this happens?

To find out, we can imagine a completely hypothetical, bare-bones 5-line C program. This represents a classic vulnerable snippet—throwing out edge cases and error handling entirely so we can just focus on the raw mechanism. The goal is to theoretically observe what happens to the operating system when a compiled binary tries to execute a basic string payload.

#include <stdlib.h>

int main() {
    system("echo hello; touch /tmp/pwned");
    return 0;
}

I learn best by observing the physical reality of the system, so if we were to compile this little file and hypothetically run it through strace -f with the -e trace=execve flag, the terminal would spit out the literal steps! And we absolutely have to add that -f flag to strace! (If you forget -f, strace only watches the main process and completely misses everything the child processes do, which is usually the part you actually care about when processes spawn other processes!)

execve("./a.out", ["./a.out"], 0x7ffd5...) = 0
[pid 12345] execve("/bin/sh", ["sh", "-c", "echo hello; touch /tmp/pwned"], 0x7ffd5...) = 0
[pid 12346] execve("/usr/bin/echo", ["echo", "hello"], 0x560f...) = 0
[pid 12347] execve("/usr/bin/touch", ["touch", "/tmp/pwned"], 0x560f...) = 0

Okay, let's break down this trace output line-by-line, because this is where the magic (and the danger) happens.

• Line 1: The kernel receives the command to start the compiled a.out program. This is just normal initialization—when you run a binary, the operating system uses execve to load it into memory. Nothing scary yet!
• Line 2: This line holds the breakthrough realization! The system() function acts as a completely gullible middleman. It literally spawns /bin/sh -c and hands that shell process the entire raw string (echo hello; touch /tmp/pwned).
• Lines 3 & 4: The standard system() library function manual page actually confirms this exact behavior, but seeing it laid out in an strace simulation makes it so much clearer. We can physically see the shell process fork twice to execute echo and touch as entirely separate, isolated commands.

So, why is this so bad? Because /bin/sh is primarily a text parser! It reads the raw string from left to right. Upon hitting the semicolon, the shell assumes it has found two independent instructions rather than understanding the original programmer's intent for a single safe echo command. It blindly respects the punctuation. This exact parsing behavior—where data is accidentally interpreted as control characters because of a simple string concatenation—directly causes OS Command Injection vulnerabilities.

But we can contrast this dangerous middleman behavior by looking at the safe alternative: using the execve syscall directly. This is what modern libraries and frameworks (like Python's subprocess or Rust's Command) are supposed to use under the hood when they implement a secure subprocess module. The execve function forces you to pass a perfectly organized array of memory pointers, completely bypassing the shell.

If the same vulnerable program were rewritten to use execve directly, the underlying arguments would look like this rigid array: ["echo", "hello; touch /tmp/pwned"].

Anyway, because execve passes this rigid array directly to the kernel without a shell parser anywhere in the chain, the operating system treats the second array item as a literal string argument to pass to echo. The kernel doesn't care about semicolons or spaces or special characters! It physically cannot execute the touch command because the injection vector literally does not exist. The echo program would just happily print out the literal words hello; touch /tmp/pwned to your screen :)

You can picture this as a side-by-side conceptual diagram. On the left side, system() passes a single long string to a cartoon /bin/sh character, which happily slices it up at the semicolon. On the right side, execve() passes a rigid, secure box of array slots directly to the kernel, with absolutely no shell in the middle to misinterpret the data. It's so much simpler than I thought it was!

There is a lot I still don't know about how C handles memory here, and I don't totally understand how the environment variables get safely passed down to /bin/sh during the fork step (I left out execve's envp argument completely because I didn't want to deal with it!). If you know a good way to trace the environment block during a fork without drowning in a million lines of strace output, please tell me!