I turned on HTTP/3 for a CloudFront distribution, saw the load times improve, and then did what I usually do when something works without me understanding why: I went looking for the packets.

This post has three parts. First, how HTTP got here — because HTTP/3 makes very little sense without the twenty years of workarounds that preceded it. Second, the theory, kept short. Third, the practical part: enabling it on a CDN, then capturing real QUIC traffic on my OpenWrt router with tcpdump and reading what comes back.

Part 1 — How We Got Here

HTTP/0.9 to HTTP/1.1: one file at a time

HTTP/0.9 (1991) was a single line: GET /page.html. No headers, no status codes, no content types. The server returned HTML and closed the connection.

HTTP/1.0 (RFC 1945, 1996) added headers, status codes and content negotiation. It still opened a fresh TCP connection per request, which meant paying the TCP handshake for every image on a page.

HTTP/1.1 (RFC 2068 in 1997, then RFC 2616, and today RFC 9110–9112) introduced persistent connections and Host, which is what made virtual hosting possible. It also introduced pipelining — sending several requests without waiting for each response.

Pipelining was the right idea and a practical failure. Responses had to come back in request order, so one slow response blocked everything behind it. Combined with proxies that mishandled it, browsers ended up disabling pipelining by default. Instead they opened six-ish parallel connections per origin, and web developers invented sprite sheets, domain sharding and file concatenation — all workarounds for a protocol that could only do one thing at a time per connection.

SPDY and HTTP/2: multiplexing, with a catch

Google shipped SPDY in 2009: multiple concurrent streams over one TLS connection, header compression, server push. It worked well enough that the IETF used it as the basis for HTTP/2 (RFC 7540, 2015; since replaced by RFC 9113).

HTTP/2 made the protocol binary and properly multiplexed. Many requests share one connection, interleaved as independent streams, with HPACK compressing headers. The sprite sheets and sharding became unnecessary.

But HTTP/2 still runs on TCP, and TCP delivers one ordered byte stream. The kernel will not hand your application byte 5,000 until byte 4,000 has arrived. So if a single segment is lost, every multiplexed stream stalls waiting for that retransmission — even streams whose data already arrived safely. HTTP/2 removed head-of-line blocking at the HTTP layer and left it sitting at the transport layer.

This is the problem HTTP/3 exists to solve.

QUIC and HTTP/3

Fixing it inside TCP was not realistic. TCP lives in the kernel, so changes ship at the speed of OS upgrades, and the internet is full of middleboxes — NATs, firewalls, “optimizers” — that inspect TCP headers and quietly break anything unfamiliar. That resistance to change is usually called protocol ossification.

So Google built gQUIC (around 2012), a new transport on top of UDP. UDP was not chosen for speed. It was chosen because it is the one thing middleboxes reliably pass through without opinions, and because it can be implemented in userspace — so a browser can ship transport improvements on its own release cycle.

The IETF took the idea and rebuilt it as a standard. QUIC (RFC 9000, May 2021) is the transport, with TLS 1.3 integrated rather than layered on (RFC 9001) and its own loss recovery (RFC 9002). HTTP/3 (RFC 9114, June 2022) is HTTP mapped onto QUIC.

Worth noting: QUIC originally stood for “Quick UDP Internet Connections”, but the IETF dropped the expansion. It is just a name now.

Part 2 — The Theory, Briefly

Where HTTP/3 sits

The biggest conceptual shift is that the transport moved into userspace and TLS stopped being a separate layer.

HTTP/1.1 & HTTP/2HTTP/3HTTP/1.1 · HTTP/2TLS 1.2 / 1.3encryption onlyTCPordered byte streamIPHTTP/3QUICstreams · loss recovery · TLS 1.3UDPindependent datagramsIPtransport lives in the kerneltransport lives in userspace

TCP and TLS were separate layers that each needed their own handshake. QUIC merges them: the transport handshake and the cryptographic handshake are the same exchange.

Fewer round trips

With TCP you pay one round trip for the TCP handshake, then another for TLS 1.3, before the first byte of the request goes out. QUIC combines them into one. On a repeat visit it can attach the request to the very first packet — 0-RTT.

HTTP/2 — TCP + TLS 1.32 round trips before the request is sentclientserverSYNSYN-ACKACK + ClientHelloServerHello + FinishedHTTP/3 — QUIC, first visit1 round trip — transport and crypto in one exchangeInitial — ClientHelloInitial + Handshake — ServerHello, keysHTTP/3 — QUIC 0-RTT, returning visitrequest rides along with the first packetInitial + 0-RTT — ClientHello + GET /

0-RTT has a real caveat: those early packets are replayable by an attacker who captures them, so they must only carry idempotent requests. This is why CDNs typically restrict 0-RTT to GET and HEAD.

Head-of-line blocking, actually fixed

This is the part that matters most.

HTTP/2 over TCP — one ordered byte streamstream ALOSTstream Bstream Cstream BB and C arrived intact — but TCP holds them until the gap is retransmittedHTTP/3 over QUIC — independent streamsstream ALOSTstream Bstream Cstream BB and C are delivered immediately — only stream A waits for its retransmissionQUIC tracks delivery per stream, so loss is contained to the stream it affected

QUIC gives each stream its own delivery guarantee. A lost packet only blocks the stream it belonged to. On a clean wired connection the difference is modest; on lossy mobile networks it is substantial, which is where most of the real-world gains come from.

Connection IDs and migration

A TCP connection is identified by the four-tuple of source IP, source port, destination IP and destination port. Change your IP — walk out of Wi-Fi range onto cellular — and the connection is dead.

QUIC identifies connections by a connection ID carried in the packet itself. The addresses can change underneath and the session survives. Your video keeps playing when your phone switches networks.

A couple of details worth knowing

Header compression changed. HTTP/2’s HPACK assumes headers arrive in order — which is exactly the assumption QUIC breaks. HTTP/3 uses QPACK, which achieves similar compression without requiring ordered delivery.

Clients don’t start with HTTP/3. There is no UDP equivalent of “just connect and see”. A browser normally connects over HTTPS first and the server advertises support with an Alt-Svc header:

alt-svc: h3=":443"; ma=86400

The client remembers this and tries QUIC next time. A HTTPS resource record in DNS can advertise alpn="h3" and skip that first round entirely. If UDP is blocked, clients fall back to HTTP/2 — usually invisibly, sometimes after an awkward timeout.

Part 3 — Practice

Turning it on at the CDN

On CloudFront this is a single setting. The distribution’s supported protocol versions become HTTP/3 (or HTTP/2 and HTTP/3), and CloudFront handles QUIC termination at the edge:

aws cloudfront get-distribution-config --id <DISTRIBUTION_ID> > dist.json
# set "HttpVersion": "http2and3" in the DistributionConfig, then:
aws cloudfront update-distribution \
  --id <DISTRIBUTION_ID> \
  --distribution-config file://updated-config.json \
  --if-match <ETag>

Two things worth understanding before you flip it:

  • This is edge-only. QUIC terminates at the CDN. The origin fetch is still whatever it was — usually HTTP/1.1 or HTTP/2 over TCP. You are improving the client-to-edge leg, which is the leg with the variable latency and the packet loss, so that is the right leg to improve.
  • It is a negotiation, not a switch. Clients that can’t do HTTP/3 keep using HTTP/2. Nothing breaks; some clients just don’t benefit.

Verify from the client side rather than trusting the console:

# Does the response advertise h3?
curl -sI https://example.cloudfront.net/ | grep -i alt-svc
# alt-svc: h3=":443"; ma=86400

# Force HTTP/3 (needs a curl built with HTTP/3 support)
curl -sI --http3-only https://example.cloudfront.net/ | head -1
# HTTP/3 200

In the browser, add the Protocol column in the Network tab — it will show h3 once the client has picked it up. Remember the first page load may still be h2, because that is the request that learns about Alt-Svc.

Watching it on my router

The part I actually enjoyed. My home router runs OpenWrt, which means it is a real Linux box with tcpdump available — so I can watch traffic for the whole network from the one place everything passes through, without touching any client.

Install it if it isn’t there:

opkg update && opkg install tcpdump

Then capture QUIC to a CloudFront edge. HTTP/3 is UDP on port 443, so the filter is refreshingly simple:

tcpdump -i eth0 -nn -vvv 'ip6 and host 2001:db8:4f60:2b00::11e8:6c72 and port 443'

Here is what came back — addresses replaced with documentation-range equivalents, everything else untouched:

07:08:36.978339 IP6 (flowlabel 0xb0b00, hlim 63, next-header UDP (17) payload length: 494)
  2001:db8:85a3:1f2e::7c4a:9d13.60394 > 2001:db8:4f60:2b00::11e8:6c72.443: [udp sum ok] UDP, length 486
07:08:37.012490 IP6 (class 0x03, flowlabel 0x81151, hlim 57, next-header UDP (17) payload length: 51)
  2001:db8:4f60:2b00::11e8:6c72.443 > 2001:db8:85a3:1f2e::7c4a:9d13.60394: [udp sum ok] UDP, length 43
07:08:37.014943 IP6 (flowlabel 0xb0b00, hlim 63, next-header UDP (17) payload length: 61)
  2001:db8:85a3:1f2e::7c4a:9d13.60394 > 2001:db8:4f60:2b00::11e8:6c72.443: [udp sum ok] UDP, length 53
07:08:37.119436 IP6 (class 0x02, flowlabel 0x81151, hlim 57, next-header UDP (17) payload length: 8004)
  2001:db8:4f60:2b00::11e8:6c72.443 > 2001:db8:85a3:1f2e::7c4a:9d13.60394: [bad udp cksum 0x52cf -> 0xfb63!] UDP, length 7996
07:08:37.119437 IP6 (class 0x02, flowlabel 0x81151, hlim 57, next-header UDP (17) payload length: 16687)
  2001:db8:4f60:2b00::11e8:6c72.443 > 2001:db8:85a3:1f2e::7c4a:9d13.60394: [bad udp cksum 0x65e6 -> 0xf617!] UDP, length 16679
07:08:37.119523 IP6 (class 0x02, flowlabel 0x81151, hlim 57, next-header UDP (17) payload length: 25285)
  2001:db8:4f60:2b00::11e8:6c72.443 > 2001:db8:85a3:1f2e::7c4a:9d13.60394: [bad udp cksum 0xe209 -> 0x12fa!] UDP, length 25277
07:08:37.127947 IP6 (flowlabel 0xb0b00, hlim 63, next-header UDP (17) payload length: 61)
  2001:db8:85a3:1f2e::7c4a:9d13.60394 > 2001:db8:4f60:2b00::11e8:6c72.443: [udp sum ok] UDP, length 53

Reading it

It really is just UDP. No SYN, no Flags [S.], no handshake to watch. tcpdump reports UDP, length N and nothing about streams or requests, because there is nothing else it can see. Everything above UDP is QUIC, and QUIC encrypts not only the payload but most of its own header — packet numbers are protected too. The connection ID is visible; the rest is opaque. That is the design working as intended, and it is why passive network monitoring tells you much less about HTTP/3 than it did about HTTP/1.1.

Response sizes that can’t be real. UDP, length 25277 on a link with a ~1500 byte MTU is not a datagram that crossed the wire. Those are the kernel’s receive offload (GRO) merging many QUIC datagrams into one before the capture hook sees them. The bad udp cksum on exactly those oversized packets is the same story — the checksum belongs to the original datagrams, not to the synthetic merged one, so tcpdump recomputes it and disagrees. Both are capture artifacts, not network problems. The genuinely small packets a few lines up all report udp sum ok.

If you want to see the datagrams as they actually arrive, disable offload for the capture:

ethtool -K eth0 gro off lro off   # remember to turn it back on

The traffic class byte carries congestion signals. QUIC uses ECN, and you can read it here. The class field is DSCP in the top six bits and ECN in the bottom two. class 0x02 is ECT(0) — “this sender supports ECN”. class 0x03 is CE, congestion experienced: a router along the path marked that packet instead of dropping it. Note the client’s packets show no class at all, meaning Not-ECT — the asymmetry is normal, since each direction negotiates independently.

Hop limits tell you where you are. The CDN’s packets arrive with hlim 57 — the IPv6 hop limit after crossing the internet, so roughly seven hops away. Outbound packets show hlim 63, one less than the typical starting value of 64, because the router had already decremented them by the time they hit the capture point.

The flow label is stable per direction. flowlabel 0xb0b00 outbound, 0x81151 inbound, constant across the exchange. That is deliberate — it lets routers keep a flow on a consistent path without inspecting anything above IP.

If you want to see inside

tcpdump cannot decrypt QUIC, but you can hand Wireshark the keys. Launch a browser with a key log file, capture in parallel, and point Wireshark at it:

SSLKEYLOGFILE=/tmp/keys.log chromium https://example.cloudfront.net/

Set that path in Preferences → Protocols → TLS → (Pre)-Master-Secret log filename and Wireshark will decode QUIC frames, streams and HTTP/3 headers properly.

Operational notes

A few things worth knowing before enabling this on something you’re responsible for:

  • UDP/443 has to be open outbound. Plenty of corporate networks allow TCP/443 and drop everything else. Those users silently fall back to HTTP/2, which is fine — but it means “we enabled HTTP/3” and “our users are on HTTP/3” are different claims. Measure the second one.
  • Your monitoring gets quieter. Anything that inferred request behaviour from TCP-level telemetry sees far less now. Move that visibility to the CDN’s own logs, which record the protocol per request.
  • UDP can be treated as second-class. Some middleboxes rate-limit or deprioritise UDP, and QUIC is more CPU-hungry than TCP because the transport runs in userspace without decades of kernel optimisation behind it.
  • Measure the leg you changed. Improvement shows up in connection setup and in loss-heavy conditions. Compare like for like — same regions, same clients, ideally the same users before and after — rather than comparing an aggregate that mixes mobile and wired traffic.

Closing

The history explains the design. Pipelining failed, so we opened six connections. Six connections were wasteful, so HTTP/2 multiplexed them — and hit TCP’s ordering guarantee. TCP couldn’t be changed, so QUIC was built on UDP instead, where it could be.

Enabling it is a checkbox. Understanding what that checkbox does turned out to be a much better use of an evening — and the packets are right there on the router if you want to look.

References