Reverse Proxy / Load Balancer

MezHub can be deployed behind a TLS-terminating reverse proxy or load balancer. This is the recommended production topology — the load balancer owns the public TLS certificate, terminates client traffic at the edge, and gives you one place to apply rate limiting and DDoS protection. It fronts a single MezHub instance: an agent's reverse tunnel lives in one MezHub process, so balancing across several instances would strand connections on an instance that holds no tunnel for the requested node.

Architecture

When deployed behind a load balancer, the traffic flow is:

Client                    Load Balancer               MezHub
  |                           |                          |
  |---[TLS]--- HTTPS :443 --->|---[plaintext]--- :3080 ->|  Web UI + API
  |---[TLS]--- SSH :3023 ---->|---[TCP]--------- :3023 ->|  SSH proxy
  |---[TLS]--- Tunnel :3024 ->|---[TCP]--------- :3024 ->|  Agent reverse tunnels
  |---[TLS]--- gRPC :3025 --->|---[h2c]--------- :3025 ->|  Auth gRPC API

The load balancer terminates TLS on the public ports and forwards cleartext upstream to MezHub. MezHub's gRPC auth listener (port 3025) is the only port that needs an explicit toggle to accept cleartext: set MEZITE_GRPC_ALLOW_HTTP=true (orauth.grpc_allow_http: true in YAML) to wrap the gRPC server in an h2c handler. To skip mTLS on the gRPC port entirely (e.g. when the LB terminates TLS without re-encrypting), also setMEZITE_AUTH_H2C=true. The proxy's HTTPS listener (3080) can be flipped to cleartext via proxy.tls_mode: "off" in YAML. Clients still see TLS — the LB handles the encryption on the public side.

h2c vs single-port ALPN — two different things

Two distinct features of the proxy are easy to confuse:

• Auth gRPC h2c(MEZITE_AUTH_H2C / MEZITE_GRPC_ALLOW_HTTP) — Controls the internal :3025 auth gRPC listener. Used when a TLS-terminating load balancer fronts mezhuband forwards cleartext upstream. This is described below in the"Why h2c for gRPC?" section.
• Single-port ALPN routing(proxy.single_port / MEZITE_SINGLE_PORT) — Controls how external clients reach the proxy. When on, SSH, agent tunnel, and HTTPS traffic all arrive on the proxy's HTTPS listener and are demultiplexed by the TLS ALPN protocol the client negotiates. Used when you want to expose a Mezite cluster entirely on :443. Single-port mode does notimply or require h2c — these are independent knobs.

If the load balancer is also doing TLS termination for the public listeners, choose carefully: single-port mode wants TLS passthrough all the way to mezhub so the proxy can read the ALPN protocol itself; a TLS-terminating LB in front of single-port mode would strip ALPN and break SSH/tunnel routing.

Why h2c for gRPC?

gRPC requires HTTP/2. When a load balancer terminates TLS, the connection between the load balancer and MezHub is plaintext. Standard HTTP/2 requires TLS (via ALPN negotiation), so plaintext HTTP/2 needs a special mode called h2c (HTTP/2 cleartext).

Two env vars control the gRPC port:

• MEZITE_GRPC_ALLOW_HTTP=true — wraps the gRPC server in an h2c handler so it accepts h2c prior-knowledge(direct gRPC clients that send HTTP/2 without negotiation) andHTTP/1.1 Upgrade: h2c (reverse proxies that upgrade). mTLS credentials are still loaded.
• MEZITE_AUTH_H2C=true — additionally disables TLS on the gRPC port (the listener becomes plaintext). Use this when the LB terminates TLS and you do not want re-encryption between the LB and MezHub.

For most reverse-proxy deployments, setMEZITE_GRPC_ALLOW_HTTP=true; addMEZITE_AUTH_H2C=true only if the LB does not re-encrypt upstream.

MezHub Configuration

cluster_name: my-cluster
data_dir: /var/lib/mezite

database:
  driver: sqlite          # or: postgres
  url: /data/mezhub.db    # or: postgres://user:pass@host/mezite

auth:
  enabled: true
  listen_addr: 0.0.0.0:3025

proxy:
  enabled: true
  listen_addr: 0.0.0.0:3080
  ssh_listen_addr: 0.0.0.0:3023
  tunnel_listen_addr: 0.0.0.0:3024

  # The public address that clients use to reach this cluster.
  # This must match the hostname that resolves to your load balancer.
  public_addr: mezite.example.com

  # Disable TLS on the proxy — the load balancer handles TLS.
  tls_mode: "off"

# Enable h2c (HTTP/2 cleartext) on the gRPC auth port (3025).
# This is the targeted option — mTLS credentials stay loaded.
MEZITE_GRPC_ALLOW_HTTP=true

# Add this only if the LB terminates TLS and does NOT re-encrypt
# upstream. It disables TLS on the gRPC port entirely.
# MEZITE_AUTH_H2C=true

To trust an HTTP header injected by the LB (for rate limiting and theclient_ip audit field), setproxy.trusted_ip_header in mezite.yaml — this knob is YAML-only, there is no env-var binding for it. The header only applies to the HTTPS proxy listener; for SSH, tunnel, and any L4 passthrough setup, use PROXY protocol (next section).

proxy:
  trusted_ip_header: X-Forwarded-For   # or: Fly-Client-IP, CF-Connecting-IP, etc.

Preserving the real client IP — PROXY protocol v2

proxy.trusted_ip_header only works on HTTP listeners: it reads an HTTP header the load balancer injects. For the SSH, tunnel, and (in single-port mode) gRPC listeners there is no HTTP layer, so an HTTP header can't carry the source IP. Any deployment where the load balancer operates in TLS passthrough / L4 mode (e.g. AWS NLB with TCP targets, HAProxy in mode tcp) hits the same limit even on the HTTPS port: the LB never sees cleartext, so it can't add headers.

MezHub's listeners support PROXY protocol v1 and v2— a binary header the LB prepends to each TCP connection that carries the original client address. It survives TLS passthrough because it sits in front of the TLS handshake rather than inside HTTP. Enable it on the listener and configure your LB to send it on every port you expose through MezHub.

proxy:
  proxy_protocol: "on"

  # Only peers whose direct TCP source IP falls in one of these CIDRs
  # are allowed to send PROXY headers. Keep this as tight as the LB's
  # actual egress range — any host inside the CIDR that can reach the
  # listener could otherwise forge client_ip in the audit log by
  # sending a crafted PROXY header.
  proxy_protocol_trusted_cidrs:
    - 10.0.0.0/24    # example: the LB's private subnet
    - 192.0.2.5/32   # example: a single-IP LB

Peers whose source IP is NOT in the trusted list are rejected if they send a PROXY header — so an attacker who can reach the port directly can't spoof client_ip. Trusted peers that send a connection without a PROXY header fall through to the raw TCP address, which means in-cluster health probes (kubelet, etc.) continue to work when you flip the flag on.

Once enabled, every downstream code path that previously saw the LB's IP — audit event client_ip, the per-IP connection limiter, role IP pinning — transparently sees the real caller. No code or query changes needed.

Port Mapping

Your load balancer needs to forward traffic to MezHub's internal ports. The mapping depends on whether you expose each service on its standard port or consolidate onto fewer external ports.

Standard Port Mapping

Agent Configuration

Agents connecting through a TLS-terminating load balancer needMEZITE_TLS_WRAP=true. This wraps the tunnel connection in TLS so the load balancer can terminate it, then the SSH tunnel protocol runs over the plaintext connection inside.

# Connect to the load balancer's tunnel port
MEZITE_PROXY_ADDR=mezite.example.com:3024

# Auth gRPC address (through the load balancer)
MEZITE_AUTH_ADDR=mezite.example.com:3025

# Wrap the tunnel in TLS for the load balancer
MEZITE_TLS_WRAP=true

# Standard agent config
MEZITE_JOIN_TOKEN=<your-join-token>
MEZITE_NODE_NAME=web-server-01
MEZITE_DATA_DIR=/var/lib/mezite

Without MEZITE_TLS_WRAP=true, the agent sends raw SSH protocol on the tunnel port. The load balancer wouldn't know how to handle this (it expects TLS on the external port). With TLS wrapping, the agent sends TLS on the wire, the load balancer terminates it, and the plaintext SSH tunnel protocol reaches MezHub.

Client Configuration

Clients (msh, mezctl) connect to the load balancer's external address. Always give --proxy an explicit host:port: when the port is omitted,msh assumes :3080, which is MezHub's internal port, not the :443 your load balancer listens on.

# Login — the LB listens on :443 and forwards to MezHub :3080.
# Name the port: bare "mezite.example.com" would resolve to :3080,
# which the load balancer is not listening on.
msh login --proxy=mezite.example.com:443 --user=admin

# SSH (connects to LB on :3023, LB forwards to :3023).
# The proxy address comes from the profile saved at login.
msh ssh --login=ubuntu web-server-01

# Admin CLI (connects to LB on :3025, LB forwards h2c to :3025)
mezctl --auth-server=mezite.example.com:3025 nodes list

Nginx Example

The two server blocks below belong inside nginx'shttp context; the stream block is a sibling of http at the top level of nginx.conf, not nested inside it. On nginx 1.25.1 and later, replace thehttp2 parameter on the listen directive with a separatehttp2 on; directive — the parameter form is deprecated.

# HTTPS → MezHub proxy (Web UI + API)
server {
    listen 443 ssl;
    server_name mezite.example.com;

    ssl_certificate /etc/ssl/mezite.crt;
    ssl_certificate_key /etc/ssl/mezite.key;

    location / {
        proxy_pass http://mezhub:3080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support (for web terminal)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

# gRPC → MezHub auth (port 3025)
server {
    listen 3025 ssl http2;
    server_name mezite.example.com;

    ssl_certificate /etc/ssl/mezite.crt;
    ssl_certificate_key /etc/ssl/mezite.key;

    location / {
        grpc_pass grpc://mezhub:3025;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

# SSH proxy (port 3023) — TCP passthrough with TLS termination
stream {
    upstream mezhub_ssh {
        server mezhub:3023;
    }
    server {
        listen 3023 ssl;
        ssl_certificate /etc/ssl/mezite.crt;
        ssl_certificate_key /etc/ssl/mezite.key;
        proxy_pass mezhub_ssh;
    }

    # Agent tunnel (port 3024) — TCP passthrough with TLS termination
    upstream mezhub_tunnel {
        server mezhub:3024;
    }
    server {
        listen 3024 ssl;
        ssl_certificate /etc/ssl/mezite.crt;
        ssl_certificate_key /etc/ssl/mezite.key;
        proxy_pass mezhub_tunnel;
    }
}

Troubleshooting

Agent tunnel fails with "SSH handshake: EOF"

The agent connected to the wrong port. Make sure MEZITE_PROXY_ADDRpoints to the tunnel port (3024), not the HTTPS port (443). The HTTPS port serves the Web UI and API — it does not speak the SSH tunnel protocol.

mezctl returns "connection refused" or TLS errors

mezctl connects directly to the gRPC port (3025). Make sure your load balancer exposes port 3025 and forwards it as h2c (HTTP/2 cleartext) to MezHub's internal port 3025.

Agent recording upload fails with 502

The agent uploads recordings via gRPC on port 3025. If your load balancer does not support HTTP/2 on this port, the upload will fail. Ensure the load balancer is configured for gRPC/h2c on port 3025 and set MEZITE_GRPC_ALLOW_HTTP=true on MezHub so its gRPC server accepts h2c (add MEZITE_AUTH_H2C=true as well if the LB does not re-encrypt upstream).

Next Steps

• SSH Access — Set up agents and connect via SSH.
• Session Recording — Configure recording modes and storage.
• Configuration Reference — Full list of environment variables and config options.