Why We Reverted Our AWS Fargate Migration on Mezite Cloud

In May, we moved Mezite Cloud tenant workloads to AWS Fargate and then moved them back.

The attraction was real. Fargate gives each pod its own compute boundary and removes much of the work involved in managing node groups. We accepted a trade-off: lose the Kubernetes NetworkPolicy enforcement available on our EC2 nodes, gain a dedicated kernel for each pod, and use Security Groups for Pods as the network boundary.

Then we tested the boundary from one tenant workload to another. The connection succeeded.

That result did not make Fargate insecure, and it did not make per-pod virtual machines worthless. It showed that compute isolation and network isolation answer different questions. For our shared-cluster threat model, we needed both a strong application identity layer and an independently enforced Layer 3/4 boundary. The security-group-only design we had deployed did not provide the latter, so we reverted it.

The Requirement We Were Testing

Kubernetes allows pod-to-pod traffic by default. In a multi-tenant cluster, we do not want a compromised tenant workload to gain network reachability to another tenant simply because both workloads share a cluster.

Our tenant policy therefore denies traffic by default and adds only the paths the workload needs. The production policy is named tenant-isolation and selects every pod in its tenant namespace with podSelector: {}. Its ingress rules permit traffic from the namespace carrying the role=ingress label, traffic within the same tenant namespace, and a narrowly scoped platform-monitoring path. Its egress rules separately allow DNS, workload identity, explicitly configured service endpoints, and the public internet while excluding private and link-local ranges.

This abbreviated excerpt shows the important shape. It is not a complete deployment manifest: the narrow platform dependency rules are deliberately omitted.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tenant-isolation
namespace: tenant-example
spec:
podSelector: {} # Select every pod in this tenant namespace
policyTypes:
  - Ingress
  - Egress
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            role: ingress
      - podSelector: {} # Same-namespace traffic
egress:
  # Separate narrow rules allow required platform dependencies.
  - to:
      - ipBlock:
          cidr: 0.0.0.0/0
          except:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
            - 169.254.0.0/16

The link-local exclusion is important. Allowing 0.0.0.0/0 while excluding only RFC 1918 ranges would still leave endpoints such as the EC2 Instance Metadata Service reachable. Where a workload needs a link-local platform endpoint, we add a separate rule limited to that endpoint and port rather than reopening the whole range.

NetworkPolicy is not our only control. Tenant traffic is authenticated, application permissions are enforced above the network layer, and workloads are hardened and spread across nodes where capacity permits. NetworkPolicy removes reachability that the application does not need; it does not make compromise “impossible” or remove the shared-kernel risk of EC2-backed nodes.

What Fargate Supports—and What It Does Not

AWS added native Kubernetes NetworkPolicy enforcement to the Amazon VPC CNI in 2023. That announcement applied to supported EC2 node configurations, not to every EKS compute type. The current EKS documentation still states that VPC CNI network policies apply only to Amazon EC2 Linux nodes and not to Fargate or Windows nodes.

The implementation explains the boundary. On supported EC2 nodes, the VPC CNI network-policy agent programs eBPF rules on each node. Fargate does not support DaemonSets or customer access to the host, so that node-agent enforcement model is unavailable there. Fargate does support assigning different VPC security groups to individual pods, but security groups and Kubernetes NetworkPolicies are not interchangeable controls.

Our failed probe exposed that distinction. The Fargate pods needed cluster connectivity in addition to their tenant-specific security-group rules, and the resulting security-group set still allowed a cross-tenant path. We could have designed more AWS infrastructure around that constraint, but it would have created a second policy model whose selectors, reconciliation, quotas, and failure modes differed from the Kubernetes resources that define the workloads.

We chose the boundary we could express, test, and reconcile in one place.

What We Run Now

Tenant workloads now run on EKS managed node groups backed by EC2. We enable Amazon VPC CNI NetworkPolicy support and use its eBPF-based enforcement. Policy decision logs are available from the network-policy agent, giving us evidence for both allowed and denied probes instead of relying on the presence of a YAML object in the Kubernetes API.

We run the VPC CNI in NETWORK_POLICY_ENFORCING_MODE=standard, not its strict startup mode. Strict mode temporarily denies all traffic while policy is being programmed, but it also requires every workload that starts in the cluster to have suitable policy coverage. Enabling it before that coverage existed disrupted shared infrastructure workloads.

In standard mode, a new pod is initially allowed while the agent programs its policy. We reduce that window by creating tenant-isolation before creating the tenant Deployment, and the control plane reconciles the policy for existing tenants on restart. This is a deliberate trade-off, not an absolute guarantee. A future move to strict mode would require complete policy coverage for every namespace first.

Returning to EC2 also returned a shared host kernel to the threat model. We mitigate that risk with workload hardening, encrypted and authenticated application protocols, network isolation, and placement rules. Organizations that require the strongest workload boundary should use separate clusters or dedicated compute rather than treating a namespaced NetworkPolicy as equivalent to a hardware boundary.

The Lesson

The mistake was not trying Fargate. The mistake would have been treating a plausible architecture diagram as proof that the boundary worked.

We shipped the migration, ran an adversarial connectivity test, observed a result that contradicted our assumption, and reverted. Mezite Cloud now uses EC2-backed EKS nodes because that is where AWS currently supports the NetworkPolicy enforcement model we operate and test.

Fargate remains a strong option when its compute isolation and operational model match the workload. For our shared-cluster tenant boundary, verifiable network enforcement mattered more than removing node management.