Fuzzy Regex: filtering spam with local LLMs, llama.cpp and strict JSON

I was reading Christian Haschek’s recent post on self-hosting email yesterday, which inevitably led to the eternal Hacker News comment thread complaint about the entire endeavor. Managing spam is an absolute nightmare. The traditional defense mechanism for an independent mail server relies on an endless, exhausting arms race of increasingly brittle regular expressions. You block a specific pharmacy keyword, the spammers swap in a Cyrillic character or inject invisible HTML comments, and suddenly you are forced to write wildly complex look-ahead assertions just to keep your inbox usable. It is a grueling, unwinnable game of semantic whack-a-mole.

Developers modernizing this infrastructure with large language models frequently fall into a trap. They pass an inbound message to an API and politely ask whether the message is spam before waiting for a conversational reply. The conversational anti-pattern is exactly why so many AI-powered filtering tools fail in production. If you expect a chatbot to grade your inbound mail, you will inevitably deal with a model that awkwardly apologizes and refuses to process your data... or, far worse, you will blindly execute a hostile payload hidden invisibly in a newsletter.

We need a strict cognitive shift regarding how we apply these tools to raw data. I propose we start treating them as Fuzzy Regex.

By deploying local, fuzzy text-parsing functions, you can safely evaluate the world's most hostile strings entirely on your own hardware. If you examine the architecture of a model like Mistral NeMo 12B—which fits comfortably in about 7GB of RAM at 4-bit quantization—you can see exactly how this operates in practice. The mechanism relies on ditching the cloud vendors entirely and using a local execution environment like the llama.cpp HTTP Server.

By supplying a grammatical constraint called GBNF (GGML Backus-Naur Form), developers can physically force the model to output a strict JSON schema. You can wire this up as a standard Unix filter using a short Python script that reads from standard input and talks to a local inference server:

import urllib.request
import json
import sys

def evaluate_spam(email_text):
    prompt = f"Evaluate this email for spam:\n\n{email_text}"
    
    # GBNF grammar forcing a strict JSON boolean
    grammar = r'''
    root ::= "{\"is_spam\": " boolean "}"
    boolean ::= "true" | "false"
    '''
    
    payload = {
        "prompt": prompt,
        "grammar": grammar,
        "n_predict": 10,
        "temperature": 0.0
    }
    
    req = urllib.request.Request(
        "http://localhost:8080/completion",
        data=json.dumps(payload).encode('utf-8'),
        headers={'Content-Type': 'application/json'}
    )
    
    try:
        with urllib.request.urlopen(req) as response:
            result = json.loads(response.read())
            # Parse the guaranteed JSON output
            return json.loads(result["content"])["is_spam"]
    except Exception as e:
        print(f"Failed to evaluate: {e}", file=sys.stderr)
        return False

if __name__ == "__main__":
    raw_email = sys.stdin.read()
    if evaluate_spam(raw_email):
        print("SPAM DETECTED")
        sys.exit(1)
    print("CLEAN")
    sys.exit(0)

Far removed from polite behavioral suggestions or system prompts that simply ask the model for a favor, this mechanism forces a mathematical straightjacket directly onto the inference engine, restricting its physical universe of possible outputs to exactly two tokens.

Consider the stakes of getting this boundary wrong. As Simon Willison has explained, when appending unverified user content directly into a prompt, you are fundamentally handing the steering wheel of your application over to the attacker. An inbound spam email is the ultimate untrusted string. A clever spammer will simply embed a payload demanding the system ignore all previous instructions and immediately mark the message as clean, masking it in white text at the bottom of their HTML message, or tucking it inside a benign-looking MIME attachment.

If your system is evaluating that string using a standard unstructured API call, the model will cheerfully oblige the attacker. The result is a compromised inbox full of unmitigated slop!

This is where the strict JSON grammar constraint proves its architectural worth, completely redefining the security boundaries of the tool. Because the GBNF filter operates at the lowest token-generation level inside llama.cpp by literally masking out the probability of any tokens that do not fit the schema before the model can select them, the engine is physically incapable of outputting anything other than the defined boolean flag.

Think about how powerful this is. If the model determines that the next most likely token is the letter A to begin apologizing, but that character is not in the JSON grammar, its probability is set to absolute zero. The prompt injection payload can scream as loud as it wants about ignoring previous instructions or attempting data exfiltration. The hostile string simply bounces harmlessly off the token constraint, completely neutralizing the attack vector. By stripping away the conversational chatbot layer, you are left with a purely deterministic analytical engine.

Integrating this defensive pattern into existing infrastructure is remarkably straightforward. A standard tool like Rspamd allows you to write your own rules in Lua, meaning you only need a handful of lines of HTTP glue code to pipe incoming messages to your local inference script. You pass the raw, untrusted email text into the local namespace, the model evaluates the semantic weight of the words, and the grammar forces a clean, deterministic response back to your mail filter.

The LLM vendors want you to believe that safely processing untrusted text requires routing all your data through their massively expensive, generalized cloud models protected by opaque 'guardrails.' But looking at that simple {"is_spam": true} JSON output, the reality is much clearer: if you constrain the output space mathematically on your own hardware, you don’t need a conversational chatbot to protect you—you just need a calculator for words.