SSH Access

Mezite replaces static SSH keys with short-lived certificates issued on demand. Every connection is authenticated via a User CA and Host CA certificate pair, authorized against RBAC policies, routed through the proxy, and recorded for audit. No static keys are ever stored on servers or distributed to users.


How SSH Works Through Mezite

Traditional SSH relies on distributing public keys to ~/.ssh/authorized_keys on every server. Mezite eliminates this entirely with certificate-based authentication:

  • User CA — The Mezite auth service acts as a certificate authority. When a user logs in (via password, SSO, or MFA), Mezite issues a short-lived SSH certificate signed by the User CA. This certificate encodes the user's identity, allowed logins, and an expiration time.
  • Host CA — Each node agent receives a host certificate signed by the Host CA. Clients trust the Host CA, so there are no "unknown host" prompts and no TOFU (trust-on-first-use) vulnerabilities.
  • No static keys — Certificates expire automatically (typically within hours). There are no long-lived keys to rotate, revoke, or audit.

When you run msh ssh, the following sequence occurs:

  1. Authenticationmsh presents your user certificate (obtained during msh login) to the Mezite proxy on port 3023.
  2. Authorization — The proxy validates the certificate against the auth service. RBAC enforcement happens at session setup on the agent side using the role traits embedded in your user certificate.
  3. Routing — The proxy locates the target node via its reverse tunnel connection (established by mezd on port 3024).
  4. Session establishment — The proxy forwards the SSH session through the reverse tunnel to the agent. The agent itself terminates the SSH connection — it is the sshd for the node.
  5. Recording — The agent captures the terminal I/O stream at the PTY level and uploads it to the auth service for later playback.
Connection flow text
workstation                 Mezite proxy              target node
     |                           |                          |
     |-- msh ssh --login=user node ----->|                          |
     |   (user cert + request)   |                          |
     |                           |-- reverse tunnel ------->|
     |                           |   (proxied SSH session)  |
     |                           |                          |
     |<-- interactive session -->|<-- agent SSH session --->|
     |                           |   (recorded by agent)    |

Prerequisites

  • A running Mezite cluster (mezhub with auth and proxy services enabled).
  • The msh client CLI installed on your workstation.
  • The mezd binary available on the target node.
  • Network connectivity from the target node to the Mezite proxy on port 3024 (reverse tunnel).

Agent Setup

Generate a Join Token

Before a node can join the cluster, you need a one-time join token. Generate one with mezctl:

Generate join token bash
# Generate a token valid for 1 hour
mezctl tokens create --roles=node --ttl=1h

# Output:
# Token created: a1b2c3d4e5f6... (roles: node, expires: 2026-03-24T15:00:00Z)

Install and Configure the Agent

On the target node, install mezd and configure it to connect back to the Mezite proxy via a reverse tunnel.

Install agent bash
# Download the signed agent binary (Linux amd64)
curl -fsSL https://releases.mezite.com/latest/mezite-linux-amd64.tar.gz \
  | tar -xz -C /usr/local/bin/ mezd

Configure the agent via environment variables (e.g. in /etc/mezite/agent.env):

/etc/mezite/agent.env bash
MEZITE_AUTH_ADDR=mezite.example.com:3025
MEZITE_PROXY_ADDR=mezite.example.com:3024

# Join token (used once for initial registration)
MEZITE_JOIN_TOKEN=a1b2c3d4e5f6...

# Node metadata
MEZITE_NODE_NAME=web-server-01
MEZITE_NODE_LABELS=env=production,team=platform,os=ubuntu

Start the Agent (Reverse Tunnel)

The agent establishes a persistent reverse tunnel to the Mezite proxy on port 3024. This tunnel is how the proxy routes SSH sessions to the node, even if the node is behind a firewall or NAT.

Start the agent bash
# Start with systemd
sudo systemctl enable mezd
sudo systemctl start mezd

# Verify the agent registered
mezctl nodes ls
# HOSTNAME         TYPE   STATUS   LABELS                                   VERSION
# web-server-01    node   online   env=production,os=ubuntu,team=platform   0.1.0

Agentless OpenSSH Nodes

Some machines cannot or should not run mezd — legacy hosts, appliances, or boxes under change control. Mezite supports these as agentless OpenSSH nodes: the proxy reaches the node's own sshd directly, the node trusts the cluster's User CA via TrustedUserCAKeys, and Mezite still enforces RBAC and records sessions at the proxy.

See the dedicated Agentless OpenSSH guide for the end-to-end enrolment walkthrough, including CA-bundle export, sshd_config changes, host-key pinning, and the feature-by-feature differences vs. agent-based nodes.


Host User Auto-Provisioning

Roles with create_host_user: true let an agent provision the requested OS login on first use rather than requiring every Unix account to exist up front. The agent runs the equivalent of useradd with the right groups and (optionally) sudoers entries when the session opens.

Role: per-team auto-provisioned host users yaml
kind: role
version: v1
metadata:
  name: dev-host-user-prov
spec:
  options:
    create_host_user: true
    # 'keep' leaves the account in place after the session ends (default).
    # 'drop' removes the account when the user's last session on the host
    # closes. 'off' disables auto-provisioning for this role.
    create_host_user_mode: keep
    create_host_user_default_shell: /bin/bash
  allow:
    node_labels:
      env: dev
    logins:
      - "{{external.username}}"
    # Groups to add the host user to on creation.
    host_groups:
      - developers
      - docker
    # Optional sudoers fragments written to /etc/sudoers.d/<role>-<user>.
    host_sudoers:
      - "ALL=(root) NOPASSWD: /usr/bin/systemctl restart app"

The most-permissive create_host_user_mode across a user's roles wins (keep > drop > off). Host groups and sudoers are unioned across roles — be deliberate about which roles get sudoers entries.

Agentless nodes do not support auto-provisioning; the cluster has no agent on the box to run useradd. Plan to provision Unix accounts out of band on agentless hosts.


PAM Integration

The agent can run sessions inside a PAM service so site-specific policy (account-status checks, MOTDs, session limits, auditd hand-off) fires on every Mezite-mediated login.

Enable on the agent by setting MEZITE_PAM_SERVICE=<name>; the value is the name of a PAM service file under /etc/pam.d/. The agent opens the named service for every session it brokers and reports any PAM failure back to the client as a session-start error.

Pick a PAM service for Mezite sessions bash
# Reuse the system sshd PAM stack
sudo cp /etc/pam.d/sshd /etc/pam.d/mezite

# Trim or replace modules as needed for the Mezite path. For example,
# you might drop pam_motd if you do not want a per-session MOTD, or
# add pam_loginuid for auditd correlation:
sudo tee -a /etc/pam.d/mezite > /dev/null <<EOF
session    required     pam_loginuid.so
EOF

# Wire the agent
echo 'MEZITE_PAM_SERVICE=mezite' | sudo tee -a /etc/mezite/agent.env
sudo systemctl restart mezd

PAM applies only to agent-based nodes — agentless nodes already run PAM through their own sshd and Mezite has no in-line hook into that path.


Using msh ssh

Log In and List Nodes

Login and list nodes bash
# Log in to Mezite (obtain user certificate from the User CA)
msh login --proxy=mezite.example.com:3080 --user=alice

# List nodes registered with the cluster
msh ls
# HOSTNAME         ROLE  STATUS  LABELS                                   VERSION
# web-server-01    node  online  env=production,os=ubuntu,team=platform   0.1.0
# web-server-02    node  online  env=production,os=ubuntu,team=platform   0.1.0
# staging-01       node  online  env=staging,os=ubuntu,team=platform      0.1.0

Connect by Name

SSH by node name bash
# Connect to a specific node
msh ssh --login=ubuntu web-server-01

# Run a one-off remote command
msh ssh --login=ubuntu web-server-01 -- uptime

# Connect using the user@host shorthand
msh ssh ubuntu@web-server-01

Filter Nodes by Label

Use the --filter flag on msh ls to narrow the node list by label. --filter is repeatable and applies AND logic across keys. msh ssh itself targets a single hostname — use msh ls --filter=... to find the right node first.

Filter by label bash
# List nodes matching env=staging
msh ls --filter=env=staging

# Combine multiple filters (all must match)
msh ls --filter=env=production --filter=team=platform
# HOSTNAME         ROLE  STATUS  LABELS                                   VERSION
# web-server-01    node  online  env=production,os=ubuntu,team=platform   0.1.0
# web-server-02    node  online  env=production,os=ubuntu,team=platform   0.1.0

Specify Login User

The --login flag specifies which OS user to authenticate as on the remote node. The login must be listed in your role's logins field (the allow.logins list in the role spec).

Login flag examples bash
# Explicit login flag
msh ssh --login=deploy web-server-01

# user@host shorthand
msh ssh deploy@web-server-01

# If your role uses template variables, your Mezite username may work:
msh ssh --login=alice web-server-01

SCP File Transfers

Use msh scp to transfer files through the Mezite proxy. All transfers are authenticated with your certificate and logged in the audit trail.

File transfer with msh scp bash
# Upload a file to the remote node
msh scp ./deploy.tar.gz ubuntu@web-server-01:/tmp/

# Download a file from the remote node
msh scp ubuntu@web-server-01:/var/log/app.log ./

# Recursive directory upload
msh scp -r ./config/ ubuntu@web-server-01:/etc/app/

# Note: msh scp does not support remote-to-remote copies — one of src/dst
# must always be local.

SSH ProxyCommand Integration

If you prefer to use the native ssh client (for editor integration, Ansible, or other tooling), configure msh as a ProxyCommand. This routes your native SSH sessions through the Mezite proxy with full certificate auth and audit.

Generated ~/.ssh/config entry bash
# Generated by msh config after msh login.
Host web-server-01.production.mezite
    HostName web-server-01
    ProxyCommand msh proxy ssh %r@%h:%p
    IdentityFile ~/.mezite/profiles/<proxy>/keys/ssh_key
    CertificateFile ~/.mezite/profiles/<proxy>/keys/ssh_key-cert.pub
    UserKnownHostsFile ~/.mezite/known_hosts

# Then use native ssh:
# ssh ubuntu@web-server-01.production.mezite

Prefer the generated config for day-to-day use: it includes the right identity files and per-node host aliases, so OpenSSH validates each node's host certificate against the exact node name.

You can generate this configuration automatically:

Generate SSH config bash
# Print SSH config for every node visible from your active profile
msh config

# Or append directly to ~/.ssh/config
msh config --append

# Now use native ssh directly
ssh ubuntu@web-server-01.mezite

This is particularly useful for tools like rsync, sshfs, and Ansible that rely on the native ssh binary.


Port Forwarding

Local port forwarding (ssh -L) is gated by the port_forwarding role option. Forwarding is allowed when any of the user's roles has port_forwarding: true (the shipped admin and ssh-access roles do); it is denied only when none of them allow it, in which case the forwarding channel is refused and an access.denied.port_forwarding audit event (code T4002W) is emitted. A granted forward records a port_forwarding.start event (T4001I) with the target address, so tunnels are visible in the audit trail. Forwarded traffic is not session content and is not recorded.

The enforcement point depends on the node type. For agent (mezd) nodes the user's SSH connection is end-to-end encrypted to the agent, so the agent enforces the gate on its own channel loop. For agentless OpenSSH nodes the proxy terminates the connection (man-in-the-middle) and relays the forwarding channel to the target sshd after the same check. Both paths apply identical RBAC.

msh ssh does not register OpenSSH-style -L, -R, or -D flags. To use native port forwarding, configure msh proxy ssh as a ProxyCommand (see the section above) and run the standard ssh binary with the forwarding flags you want. This is exactly how IDE remote-development tools work — see the Remote development with VS Code guide.


Keepalives and Session Lifetime

Mezite sends SSH keepalives from the server toward the client on every leg of a connection (proxy and agent). They keep an otherwise idle session — like an editor left open over lunch — alive past the idle timeouts that load balancers and NAT gateways impose, and they detect a peer that has silently gone away so its connection can be reaped.

Config keyDefaultDescription
ssh.keep_alive_interval
MEZITE_SSH_KEEP_ALIVE_INTERVAL
5mHow often the server sends a keepalive. Deliberately under the ~350s idle timeout common to cloud load balancers so an idle session is never dropped from underneath you.
ssh.keep_alive_count_max
MEZITE_SSH_KEEP_ALIVE_COUNT_MAX
3Number of consecutive unanswered keepalives before the connection is dropped. With the defaults a dead peer is reaped after roughly 15 minutes (3 × 5m).

By default a live session survives certificate expiry — only the next connection has to re-authenticate. To force a hard cutoff at expiry, set disconnect_expired_cert: true on a role (see the Roles & RBAC reference); it OR-combines across roles. When a session is severed this way, re-run msh login and reconnect.


Session Recording and Playback

Agent SSH sessions are recorded by the agent (mezd). The agent captures terminal I/O at the PTY level — the clean text you see in your terminal, not encrypted SSH protocol bytes. Recordings are uploaded to the auth service via gRPC and stored in the configured backend (local filesystem or s3).

The recording mode is set via MEZITE_RECORDING_MODE on the agent. The default is node, which buffers the recording locally and uploads it after the session ends. Set MEZITE_RECORDING_MODE=node-sync to stream chunks to the auth service in real time instead — the agent will terminate the SSH session if the upload stream breaks.

Session playback bash
# List recent sessions
msh sessions ls
# SESSION ID                            USER   NODE          LOGIN  STARTED               ENDED
# a1b2c3d4-e5f6-7890-abcd-ef1234567890  alice  web-server-01  ubuntu 2026-04-11T10:28:35Z  2026-04-11T10:41:09Z

# Play back a session in your terminal
msh play a1b2c3d4-e5f6-7890-abcd-ef1234567890

# Play back at 2x speed
msh play --speed=2 a1b2c3d4-e5f6-7890-abcd-ef1234567890

Administrators can view all sessions; regular users can only view their own. See the Session Recording guide for recording modes, storage backends, and encryption configuration.


Advanced Configuration

Enhanced Command Capture (MEZITE_BPF_ENABLED)

In addition to the full terminal-I/O recording, the agent can emit a structured per-command stream so compliance queries like "did anyone run useradd on this host in the last 30 days?" can be answered without scanning every recording end to end. Enable on the agent with MEZITE_BPF_ENABLED=true.

Implementation note. The environment variable is named MEZITE_BPF_ENABLED for forward compatibility, but the current implementation captures commands by polling the session shell's process tree under /proc. It is not true eBPF today; very-short-lived commands ( sub-100ms) may be missed by the poll. The recording itself — the full PTY stream — always captures everything regardless of this flag. A real eBPF implementation is on the roadmap; switching backends will be transparent to operators because the audit event shape stays the same.
Enable enhanced command capture on the agent bash
MEZITE_BPF_ENABLED=true mezd start

Session Moderated Access

Status: Available — Moderated sessions are implemented. Sessions with require_session_join policies block until the required moderators join via the web API. Moderator leave terminates the session.

For sensitive environments, require a second user to observe or approve sessions in real time:

Moderated sessions role yaml
kind: role
version: v1
metadata:
  name: ssh-sensitive
spec:
  options:
    require_session_mfa: "totp"
    # require_session_join is repeatable; each entry specifies who must be
    # present and in what mode for sessions opened with this role.
    require_session_join:
      - name: require-auditor-observer
        # filter is a predicate expression evaluated against the joining user.
        filter: 'contains(user.roles, "auditor")'
        modes:
          - observer
        count: 1
        on_leave: terminate
  allow:
    node_labels:
      env: production
      sensitivity: high
    logins:
      - root

Troubleshooting

Agent fails to join

  • Verify the join token has not expired: mezctl tokens ls
  • Check network connectivity from the agent to the proxy on port 3024.
  • Review agent logs: journalctl -u mezd -f

Connection refused or timeout

  • Confirm the node appears in msh ls. If not, the agent may not be connected.
  • Check that your role grants access to the node's labels and the login you are using.
  • Verify your user certificate is valid: msh status

Permission denied

  • The login (root, ubuntu, etc.) must be listed in your role's logins field.
  • The node's labels must match your role's node_labels selector.
  • Inspect your assigned roles with mezctl users list and examine each role's allow/deny rules with mezctl roles get <name> to check for a deny rule overriding your allow rule.

Next Steps