Proxying SPDY Upgrades for Kubectl Exec
Why standard HTTP/2 breaks kubectl exec over a transparent TLS MITM proxy, and how we avoid it by staying on HTTP/1.1 and splicing the upgraded stream.
Originally published on the Iddio engineering blog. Iddio is now part of Mezite as msh local — here’s why.
When an AI agent runs kubectl exec or kubectl attach, it expects a seamless interactive session. Under the hood, this requires a persistent, bidirectional stream. Modern Kubernetes clients use WebSocket upgrades by default; older clients and fallback paths use SPDY upgrades over HTTPS.
This presents a challenge for msh local when the client routes through it. The proxy intercepts the connection, classifies the intent, evaluates policy, and then forwards the traffic. For standard HTTP requests, this is straightforward. For WebSocket or SPDY upgrades, an HTTP/2 proxy can break the connection.
Here is how we solve the protocol upgrade problem by deliberately avoiding HTTP/2 and splicing the upgraded connection.
The HTTP/2 Problem
Go’s net/http will happily negotiate HTTP/2 over TLS whenever the peer offers it. For modern web traffic, HTTP/2 multiplexing is highly desirable.
For kubectl exec, it is a trap.
The protocol upgrade relies on hop-by-hop HTTP/1.1 headers, most notably Upgrade and Connection. The HTTP/2 specification forbids connection-specific header fields. If a proxy ends up speaking HTTP/2 on either leg, those headers cannot be carried through, the API server receives a request with no upgrade headers, and the kubectl exec session fails.
The Solution: Stay on HTTP/1.1 and Splice
msh local sidesteps the problem by never offering HTTP/2 on either leg, then handling the upgrade by hand once policy has allowed it.
1. HTTP/1.1 on both legs
On the client side, the daemon terminates the intercepted TLS connection itself with tls.Server and a per-host leaf certificate. It advertises no ALPN protocols, so the client falls back to HTTP/1.1, and it reads requests with http.ReadRequest. There is no http.ResponseWriter involved, so there is nothing to hijack.
On the upstream side, the upgrade path dials the Kubernetes API server with tls.Dial and a plain tls.Config. Because no NextProtos are set, the server cannot select h2, and the connection stays HTTP/1.1.
2. Classify and decide first
The Kubernetes classifier works from the parsed request path and method. A request whose subresource is exec, attach, portforward or proxy is raised to the break-glass tier, regardless of headers. The upgrade headers play no part in classification.
Only after policy allows the request does the proxy check whether it is an upgrade request: an Upgrade header plus a Connection header containing the upgrade token. A denied request gets a 403 and is never spliced.
3. Forwarding the upgrade manually
For an allowed upgrade, the proxy opens the second TLS connection to the API server and writes the client’s request to it with req.Write, which keeps the Upgrade and Connection headers intact (they are only stripped when a request goes through an http.Transport). It then relays the server’s 101 Switching Protocols response back to the client.
4. Relaying the stream
From that point the HTTP request/response cycle is over. The proxy holds two TLS connections, one to the client and one to the API server, and starts two goroutines: one copying bytes client to server and the other server to client, until either side closes. It does not parse or modify the WebSocket or SPDY frames.
Conclusion
For requests that reach the proxy, we classify and govern the break-glass operation before the upgraded stream begins. Once authorised, we step out of the way, dropping to a plain byte relay to preserve the WebSocket or SPDY protocol semantics.
The AI agent sees a completely normal, uninterrupted kubectl exec session. We achieve the security and visibility of a transparent MITM proxy without breaking the complex protocol upgrades that critical infrastructure tools rely on.
Mezite Team
Engineering