how cloudflare rewrites your html (and how to write a proxy in go!)

I was reading a recent thread on Hacker News where people were shocked to find a random JavaScript analytics snippet—specifically https://static.cloudflareinsights.com/beacon.min.js/—appearing out of nowhere inside their static site's HTML.

The confusion made total sense to me! I actually originally thought Cloudflare functioned exclusively as a DNS provider.

But if a service just hands out IP addresses, modifying the actual website payload should be IMPOSSIBLE! Especially because modern web traffic happens over HTTPS, meaning the payload is strictly encrypted end-to-end between the visitor's browser and the origin server. If an intermediary modifies the bytes in transit, the TLS check should fail, the math breaks, and the browser should panic with a giant red warning screen.

So, how on EARTH does a network provider physically edit an encrypted HTML file in transit without the browser noticing?

It turns out it all comes down to a tiny toggle in their DNS dashboard. If you've ever set up a domain on Cloudflare, you might have noticed they have two settings for a DNS record: a "gray cloud" and an "orange cloud".

The gray cloud setting means "DNS Only". If I have a gray cloud record and run dig example.com, Cloudflare just returns my origin server's actual, real-world IP address. The HTTP traffic goes straight from the visitor's browser to my server. Cloudflare is just acting like a regular internet phonebook here, and they never touch the actual web traffic.

But the orange cloud setting means "Proxied". This is where things get wild! Whenever I'm confused about network traffic, my favorite thing to do is run a basic tool to look at the raw bytes. So instead of trying to read through a bunch of architecture diagrams to understand what "Proxied" actually does, let's just run curl -v against an orange-cloud domain and look at the TLS handshake. The -v flag is amazing because it shows you exactly what the server is offering before the HTTP data even starts flowing.

Here is what the raw output looks like:

* Connected to example.com (104.21.34.42) port 443
* ALPN, offering h2
* Server certificate:
*  subject: CN=example.com
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
> GET / HTTP/2
< HTTP/2 200 

Let's look at exactly what these bytes mean. The first major clue is the IP address 104.21.34.42 on line one. I looked this up, and it belongs to Cloudflare's massive edge network rather than the actual private server hosting the website!

But the real smoking gun is the Server certificate block! Under normal circumstances, an origin server might use a certificate issued by a recognizable authority like Let's Encrypt. Here, the issuer explicitly states O=Cloudflare, Inc..

This totally blew my mind when I first realized it: Cloudflare is terminating the TLS connection right there at the edge. By holding a valid certificate for the domain (which they automatically provision for you when you turn on the orange cloud), they act as a reverse proxy sitting directly in the middle of the conversation.

When the browser sends an encrypted request, Cloudflare’s servers decrypt it completely. Because the traffic is now sitting in plain text on their system, they can literally just parse the raw HTML response coming back from your origin server, run a quick string replacement to inject that <script> tag right before the closing </body> tag, and then re-encrypt the whole package before finally sending it down to the visitor.

Because we added a script tag, the HTML payload is physically larger. This means the proxy also has to recalculate and update the Content-Length HTTP header before sending it back to the browser. If it didn't update Content-Length, the browser would just cut the download off early, assuming it has all the bytes, and the HTML would be totally mangled.

Okay, so intermediaries have the physical capability to mess with your traffic if they terminate TLS. But what if you really don't want them to? It turns out standard HTTP rules still apply to these systems. If an origin server wants to explicitly forbid intermediaries from modifying the payload, there is a built-in mechanism for that. The server just needs to return a Cache-Control header containing the no-transform directive. Cloudflare actually respects this standard, meaning if it sees that header, it will pass the HTML through exactly as it received it without injecting the analytics beacon :)

I’m seriously amazed by how demystifying a reverse proxy feels. At the end of the day, an imposing system handling millions of requests per second is fundamentally just a program that terminates a TCP connection, reads a string, runs a quick replace function, and passes it along. It is so cool that you can see all of this happening in plain sight just by running curl and looking at the raw certificates!