Container Networking & Service Mesh: Business Guide#

What Is Container Networking?#

When you run multiple containers or services in Kubernetes, they need to find each other and communicate. Container networking is the infrastructure layer that makes this possible: assigning IP addresses to each container, routing traffic between them, and enforcing security policies about who can talk to whom.

Analogy: Container networking is like the building’s internal phone system. Every office (container) gets an extension number (IP address). The phone system (networking) routes calls between offices. Security policies decide which offices are allowed to call which others.

The default networking built into Docker and Kubernetes is minimal — enough for simple cases, but lacking security controls and visibility for production use. CNI plugins (Container Network Interface) replace this default with a more capable implementation.

What Is a CNI Plugin?#

CNI (Container Network Interface) is a standard for how Kubernetes asks a networking tool to set up networking for a new container. When Kubernetes creates a pod, it calls the CNI plugin to:

1. Assign an IP address to the pod
2. Create network routes so other pods can reach it
3. Enforce any network security policies

You install one CNI plugin per cluster. The main choices are:

• Cilium: Modern, eBPF-based, rich features, best performance at scale
• Calico: Enterprise-focused, integrates with physical network routing (BGP)
• Flannel: Simple, limited — mainly for development environments
• Weave Net: Abandoned January 2024 — do not use

What Is a Service Mesh?#

A service mesh adds a security and reliability layer on top of basic container networking. It handles things your application code shouldn’t have to:

• Encryption: All traffic between services is automatically encrypted (mTLS)
• Reliability: Automatic retries when a service temporarily fails
• Traffic management: Route 10% of traffic to a new version for testing (canary)
• Observability: Per-request latency, error rates, and dependency maps

How it works: The mesh injects a small proxy alongside each service. All traffic flows through this proxy, which handles encryption, retries, and telemetry without the application knowing. (Some newer implementations, like Istio ambient mode, skip the per-pod proxy entirely.)

When do you need it? Most small teams don’t. A service mesh makes sense when:

• Regulations require encrypted service-to-service communication (PCI-DSS, HIPAA)
• You need zero-downtime deployments with controlled traffic splitting
• You’re operating 50+ services and need systematic observability

The main service mesh tools are Istio (most features, higher complexity) and Linkerd (simpler, lighter, less features).

eBPF: Why It Matters#

Traditional container networking uses iptables — a Linux firewall system from the late 1990s. iptables was designed for simple firewalls, not for clusters with hundreds of services. In large Kubernetes clusters, iptables becomes a bottleneck.

eBPF (extended Berkeley Packet Filter) is a modern Linux kernel mechanism that replaces iptables for networking:

• Faster: Hash table lookups instead of scanning thousands of rules
• More visible: Can track every packet for detailed network monitoring
• More powerful: Can do things iptables can’t (per-connection bandwidth limits, WireGuard encryption)

Cilium is the leading eBPF-based networking solution for Kubernetes. It requires a relatively recent Linux kernel (5.10+, released in 2020).

Key Concepts Glossary#

CNI: Container Network Interface — the standard for how Kubernetes sets up pod networking.

Sidecar: A small proxy container automatically injected into every pod in a service mesh, intercepting all inbound and outbound traffic.

mTLS: Mutual TLS — both sides of a connection verify each other’s identity using certificates. Service meshes automate this.

iptables: Linux kernel’s traditional packet filtering system. Used by most Kubernetes networking until eBPF.

eBPF: Modern Linux kernel extension mechanism. Cilium uses it for faster, more capable networking.

BGP: Border Gateway Protocol — the routing protocol that powers the internet. Calico can speak BGP to integrate Kubernetes pod IPs with physical network routers.

NetworkPolicy: Kubernetes object that defines which pods can talk to which others. Like firewall rules for pod-to-pod traffic.

Envoy: The dominant proxy used inside service meshes (Istio, Consul Connect). Not a service mesh itself — it’s the engine that meshes use.

TPROXY: A Linux kernel mechanism for intercepting network traffic without modifying it. Service meshes use it to capture all traffic flowing into and out of pods.

Decision Summary#

S1 Rapid Discovery Approach: Container Networking & Service Mesh#

Scope#

This survey covers the infrastructure layer between containers and the network:

1. Container networking fundamentals — Docker network modes, CNI specification
2. CNI plugins — Calico, Flannel, Cilium, Weave Net (the Kubernetes networking ecosystem)
3. Service mesh — Istio, Linkerd, Envoy, Consul Connect, Cilium Service Mesh
4. Transparent proxying — TPROXY, network namespaces, sidecar injection mechanics

What Is Not Covered#

• Application-level networking (HTTP clients, gRPC, REST)
• Cloud provider managed networking (AWS VPC CNI, GKE dataplane v2)
• SD-WAN or physical network infrastructure

Research Method#

• Official documentation for each project
• CNCF landscape and project status pages
• GitHub repositories for star counts and activity
• Kubernetes documentation for CNI integration

Questions to Answer#

1. Which CNI plugin should a new Kubernetes cluster use in 2026?
2. When is a service mesh necessary vs. overkill?
3. What is the difference between Istio sidecar mode and ambient mode?
4. How does eBPF-based networking (Cilium) differ from iptables-based?
5. Is Weave Net still viable?

S1 Overview: Container Networking & Service Mesh#

The Container Network Stack#

Container networking adds two layers above the physical network:

Application
    ↕
Service Mesh (L7: mTLS, retries, tracing)
    ↕
Container Network Interface (L3/L4: IP routing, policies)
    ↕
Linux Kernel (iptables/nftables/eBPF)
    ↕
Physical Network

Each layer is independently configurable. A cluster can use Cilium (CNI) without any service mesh, or use Istio (mesh) on top of Flannel (CNI).

Container Networking Fundamentals#

Docker Network Modes#

Bridge mode internals: Docker creates a virtual ethernet pair (veth). One end goes into the container’s network namespace; the other connects to the docker0 bridge. iptables NAT rules translate container IP to host IP for outbound traffic.

Network Namespaces#

Linux network namespaces isolate network stacks. Each container gets its own namespace containing: interfaces, routing tables, iptables rules, sockets. The host and containers see different network views.

# List network namespaces
ip netns list

# Execute in a specific namespace
ip netns exec <ns-name> ip addr

Kubernetes pods share a network namespace across all containers in the pod (via the pause container). This is why containers in the same pod communicate over localhost.

CNI: Container Network Interface#

CNI is a specification (not a product) for how container runtimes call plugins to set up networking. When Kubernetes creates a pod:

1. kubelet creates a network namespace
2. kubelet calls the CNI plugin with ADD command + pod metadata
3. CNI plugin assigns IP, sets up routes, configures policies
4. Pod is now reachable

CNI spec: https://github.com/containernetworking/cni — simple JSON config + binary invocation. Any binary that conforms to the spec can be a CNI plugin.

Calico#

• GitHub: projectcalico/calico
• Stars: ~7,100
• Version: 3.31.3 (early 2026)
• CNCF: Not a CNCF project (Tigera-backed)
• Technology: BGP-based routing (no overlay by default); eBPF dataplane option
• Key feature: Rich NetworkPolicy support beyond Kubernetes standard; GlobalNetworkPolicy
• Use case: Production clusters needing fine-grained policy; BGP peering with physical routers

Flannel#

• GitHub: flannel-io/flannel
• Stars: ~9,400
• Version: 0.28.1
• CNCF: Not a CNCF project (CoreOS origin, now community)
• Technology: VXLAN overlay (default); also UDP, host-gw backends
• Key feature: Simplicity — minimal configuration, easy to understand
• Use case: Development clusters, simple deployments; lacks NetworkPolicy support

Important: Flannel does NOT implement NetworkPolicy. For policy enforcement, add a separate NetworkPolicy controller (e.g., Calico’s policy engine on top of Flannel).

Cilium#

• GitHub: cilium/cilium
• Stars: ~23,700
• Version: 1.19.1 (early 2026)
• CNCF: Graduated project
• Technology: eBPF — kernel programs replace iptables for packet processing
• Key features: eBPF-based (faster, more observability), Hubble network observability, built-in service mesh (Cilium Service Mesh), WireGuard encryption
• Use case: Production clusters prioritizing performance and observability; eBPF requires Linux kernel 5.10+ (full features: 5.15+)

Cilium replaces iptables entirely using eBPF programs attached to network interfaces. This reduces CPU overhead at scale and enables per-packet observability via Hubble.

Weave Net#

• GitHub: weaveworks/weave
• Status: Archived June 20, 2024 — Weaveworks shut down early 2024; repository archived read-only
• CNCF: Not a project
• Action: Do not use for new clusters; migrate existing deployments to Cilium or Calico

Service Mesh#

A service mesh adds a dedicated infrastructure layer for service-to-service communication, handling: mutual TLS (mTLS), load balancing, retries, circuit breaking, distributed tracing, and traffic management policies — without application code changes.

Sidecar pattern: Each pod gets an injected proxy container (sidecar) that intercepts all inbound/outbound traffic. The application thinks it’s talking directly to other services; actually all traffic flows through the sidecar proxies.

Envoy Proxy#

The dominant service mesh data plane (not a mesh itself). Almost all service meshes use Envoy as the sidecar proxy.

• GitHub: envoyproxy/envoy
• Stars: ~40,100
• Version: 1.36.3 (early 2026)
• CNCF: Graduated project
• Role: Receives xDS (discovery service) configuration from a control plane (Istio, etc.)
• Key features: HTTP/1.1, HTTP/2, HTTP/3, gRPC, WebSocket; rich observability; extensible via filters and WebAssembly plugins

Envoy is rarely used standalone in Kubernetes. It’s the proxy that Istio, Consul Connect, and other meshes inject as the sidecar.

Istio#

The most widely deployed service mesh. Provides a control plane that configures Envoy sidecars.

• GitHub: istio/istio
• Stars: ~37,900
• Version: 1.29.0 (early 2026)
• CNCF: Graduated project (graduated July 2023)
• Data plane: Envoy (sidecar injection into pods)
• Control plane components: istiod (Pilot, Citadel, Galley merged)
• Modes:
  • Sidecar mode: Traditional — Envoy injected into each pod
  • Ambient mode: GA in Istio 1.24 (November 2024) — No sidecars; ztunnel (Rust) node agent + optional per-namespace Envoy waypoints
• Key features: Traffic management (VirtualService, DestinationRule), mTLS, JWT auth, circuit breaking, fault injection, distributed tracing, Kiali dashboard

Ambient mode significance: Eliminates sidecar overhead (memory per pod). Ambient uses a shared ztunnel per node for L4 mTLS, with optional Envoy waypoints for L7 features. GA since Istio 1.24 (November 2024).

Linkerd#

Simpler, lighter alternative to Istio. Uses its own Rust-based proxy (not Envoy).

• GitHub: linkerd/linkerd2
• Stars: ~11,300
• Version: 2.19 (stable, Buoyant Enterprise for Linkerd) / edge-26.2.1 (open source)
• CNCF: Graduated project (July 2021)
• Data plane: Linkerd2-proxy (Rust, ~10MB per pod vs Envoy’s ~50MB)
• Key features: Automatic mTLS, HTTP/2 and gRPC load balancing, retries, timeouts, observability (Grafana dashboards built-in)
• Important: Since February 2024, stable releases are published only by Buoyant Enterprise for Linkerd (BEL); the open source channel provides edge releases only
• Use case: Clusters that want mTLS + observability without Istio complexity

Consul Connect (HashiCorp)#

Service mesh integrated with Consul service discovery.

• GitHub: hashicorp/consul
• Version: 1.22.3 (early 2026)
• License: BSL 1.1 (not OSI open source; changed from MPL 2.0 in August 2023)
• Ownership: IBM/HashiCorp (IBM acquired HashiCorp for $6.4B, completed early 2025)
• Data plane: Envoy
• Key feature: Works in Kubernetes and non-Kubernetes (VMs) simultaneously
• Use case: Mixed environments (Kubernetes + legacy VMs); already using Consul

Cilium Service Mesh#

Cilium also functions as a service mesh using eBPF, eliminating sidecars entirely.

• No sidecar injection — mesh functionality in kernel via eBPF
• mTLS: WireGuard-based node-to-node encryption
• Observability: Hubble (L3-L7 visibility)
• Trade-off: Less L7 traffic management capability vs Istio; simpler operational model

Transparent Proxying (TPROXY)#

TPROXY is a Linux kernel mechanism for intercepting traffic without modifying source/destination. Service mesh sidecars use it to capture all traffic entering/leaving a pod.

Mechanism:

1. iptables rules in the pod’s network namespace redirect all traffic to the sidecar port
2. The sidecar proxy (Envoy/Linkerd2-proxy) handles the connection
3. TPROXY preserves the original destination IP (unlike REDIRECT which rewrites it)
4. The sidecar forwards to the actual destination

# Typical Istio init container iptables rules (simplified)
iptables -t nat -A PREROUTING -p tcp -j REDIRECT --to-port 15001
iptables -t nat -A OUTPUT -p tcp -j REDIRECT --to-port 15001 -m owner --uid-owner 1337

The --uid-owner 1337 exemption prevents the sidecar’s own traffic from being redirected (avoiding infinite loops).

eBPF vs iptables#

When iptables is fine: Clusters under ~200 nodes, standard workloads. When eBPF matters: Large clusters (500+ nodes), high-throughput services, need deep observability.

Key Deprecations#

• Weave Net: Weaveworks company shutdown early 2024; repository archived June 20, 2024 (read-only)
• Consul Connect: BSL 1.1 license since August 2023; IBM acquired HashiCorp for $6.4B (completed early 2025)
• Istio sidecar-only: Not deprecated, but ambient mode (GA in v1.24, November 2024) is the recommended path for new deployments
• Linkerd stable releases: Since February 2024, stable/semver Linkerd releases require Buoyant Enterprise for Linkerd (BEL)
• kube-proxy: Cilium can replace kube-proxy entirely (kube-proxy-free mode)

S1 Recommendation: Container Networking & Service Mesh#

CNI Plugin Recommendation#

New Production Kubernetes Cluster#

Use Cilium

• CNCF graduated, eBPF-based, replaces kube-proxy, built-in observability
• Default CNI in GKE (optional), EKS (via add-on), Talos Linux, k3s (optional)
• Requires Linux kernel 5.10+; most modern distributions ship with 5.15+

Legacy Cluster or BGP-Heavy Enterprise#

Use Calico

• BGP peering with physical routers, GlobalNetworkPolicy support
• Mature (Tigera-backed), proven at scale

Development/Learning Only#

Use Flannel (or accept cluster default)

• No production use; lacks NetworkPolicy support
• Good enough for local Kubernetes (kind, minikube)

Do Not Use#

• Weave Net — company shut down January 2024; no active maintainers

Service Mesh Recommendation#

Default: No Service Mesh#

Most clusters do not need a service mesh. A service mesh is warranted when:

• You need automatic mTLS between services (regulatory compliance, zero-trust)
• You need detailed L7 observability (per-request tracing, error rates)
• You need traffic management (canary deployments, circuit breaking)

If You Need a Service Mesh#

Low complexity, mTLS + observability: Linkerd or Cilium Service Mesh

• Linkerd: automatic mTLS, Rust proxy (lightweight), built-in Grafana dashboards
• Cilium Service Mesh: no sidecars (eBPF), WireGuard mTLS, Hubble observability

Full traffic management: Istio (ambient mode)

• Use ambient mode (GA since 1.22) — no per-pod sidecar overhead
• VirtualService + DestinationRule for canary, fault injection, retries

Mixed K8s + VMs: Consul Connect

• Only option with built-in VM support alongside Kubernetes

Implementation Note#

CNI and service mesh are independent choices. You can run:

• Cilium (CNI) + Linkerd (mesh) — common, well-documented
• Calico (CNI) + Istio (mesh) — most enterprise deployments
• Cilium (CNI + mesh) — single-vendor, fewer moving parts

S1 Synthesis: Container Networking & Service Mesh#

Key Findings#

The Ecosystem Has Two Distinct Problems#

Problem 1: How do containers talk to each other? → Solved by CNI plugins Problem 2: How do services communicate reliably with security? → Solved by service mesh

These are separate concerns and separate tools. Most clusters need a CNI plugin; not all clusters need a service mesh.

CNI Plugin Landscape Has Consolidated#

Four CNI plugins dominate:

1. Cilium (eBPF, CNCF graduated) — emerging default for new production clusters
2. Calico (BGP-based, policy-rich) — dominant in enterprise clusters requiring fine-grained policy
3. Flannel (simple, no policy) — development and simple deployments only
4. Weave Net — effectively abandoned; do not use

The trend is toward Cilium. It’s the default CNI in several managed Kubernetes offerings and provides built-in observability (Hubble) that replaces separate monitoring tools.

Service Mesh: Istio Dominates But Has Complexity Concerns#

Istio is the most feature-rich and widely deployed mesh, but historically criticized for operational complexity. Ambient mode (GA 2024) addresses this by eliminating per-pod sidecar overhead and simplifying deployment.

Linkerd is the “simple” alternative: lighter proxy (Rust, ~10MB), automatic mTLS, good observability, but fewer traffic management features.

For most clusters, Linkerd or Cilium Service Mesh provides the needed mTLS + observability with lower operational overhead than Istio.

eBPF is the Direction of Travel#

The networking ecosystem is moving from iptables to eBPF:

• Cilium (CNI + mesh) is eBPF-native
• Istio ambient mode uses eBPF for some forwarding
• Linux kernel 5.10+ (widely available since 2021) makes eBPF viable everywhere

eBPF provides: better performance at scale, richer observability, ability to replace multiple tools (CNI + kube-proxy + service mesh) with a single Cilium deployment.

Weave Net is Dead#

Weaveworks shut down January 2024. Weave Net has no active maintainers. Any cluster still using Weave Net should migrate to Cilium or Calico immediately.

Decision Framework Preview#

S2 Comprehensive Analysis Approach: Container Networking & Service Mesh#

Objective#

Deep technical analysis of CNI plugins, service mesh implementations, and the transparent proxying mechanisms that underpin them.

Research Questions#

1. How does eBPF differ mechanically from iptables for packet forwarding?
2. What are the exact sidecar injection and TPROXY interception mechanisms?
3. What is Istio ambient mode’s ztunnel architecture?
4. How does Cilium’s kube-proxy replacement work?
5. What are the performance characteristics of each approach?
6. How do xDS (discovery) APIs enable dynamic proxy configuration?

Method#

• Analyzed architecture documentation for Cilium, Calico, Istio, Linkerd
• Reviewed eBPF program types relevant to networking (XDP, TC, socket)
• Examined sidecar injection admission webhooks in Kubernetes
• Reviewed CNI spec and plugin implementation patterns
• Compared benchmark data for iptables vs eBPF at scale

Files in This Phase#

• deep-analysis.md — Technical internals: eBPF, CNI plugin lifecycle, xDS, ztunnel
• synthesis.md — Key technical findings
• recommendation.md — Technical implementation guidance

S2 Deep Analysis: Container Networking & Service Mesh#

eBPF for Networking: Technical Internals#

eBPF (extended Berkeley Packet Filter) allows safely loading programs into the Linux kernel without kernel modules. For networking, eBPF programs attach to hooks in the packet path:

eBPF Program Types for Networking#

How Cilium Uses eBPF#

Cilium attaches TC programs to each pod’s veth interface (host side and pod side):

Pod Network Namespace              Host Network Namespace
 [app] --[veth-pod]-- | --[veth-host]-- [cilium TC ingress/egress programs] -- [routing]

Instead of maintaining iptables rules:

1. Each new connection lookup hits eBPF map (CT = connection tracking map)
2. Policy check hits eBPF policy map (compiled from NetworkPolicy)
3. NAT/LB decisions use eBPF LB map
4. No iptables involved for established connections

kube-proxy replacement: Cilium reads Kubernetes Service objects and programs eBPF maps for ClusterIP → pod IP load balancing. kube-proxy typically creates 20-40 iptables rules per Service; Cilium uses O(1) eBPF map lookups.

Minimum kernel versions:

• Cilium 1.14+: Requires Linux 5.10+
• Full feature set (bandwidth manager, WireGuard): Linux 5.15+
• XDP acceleration (hardware offload support): Varies by NIC driver

eBPF vs iptables at Scale#

A Kubernetes node with 100 services and 500 pods has ~5,000-10,000 iptables rules in the nat table. iptables processes rules linearly for new connections — O(n) where n = rule count. At 500 nodes × 500 pods, this creates measurable latency for new connections.

eBPF maps are hash tables with O(1) lookup. Performance does not degrade with rule count. Google’s GKE measured ~30% reduction in CPU usage for kube-proxy after switching to eBPF.

CNI Plugin Lifecycle#

When Kubernetes creates a pod, the CNI plugin is invoked:

CNI Plugin Protocol#

# Environment variables set by kubelet
CNI_COMMAND=ADD           # or DEL, CHECK, VERSION
CNI_CONTAINERID=<id>      # Container ID
CNI_NETNS=/proc/1234/ns/net  # Network namespace path
CNI_IFNAME=eth0           # Interface name in container
CNI_ARGS=K8S_POD_NAME=...   # Kubernetes metadata

# CNI config passed via stdin (JSON)
{
  "cniVersion": "0.4.0",
  "name": "k8s-pod-network",
  "type": "calico",
  "ipam": {"type": "calico-ipam"}
}

For Cilium, the ADD command:

1. Creates veth pair: cilium_veth_<podID> on host, eth0 in pod
2. Assigns IP from IPAM pool (configurable: CRD-backed, AWS ENI, host-scope)
3. Creates eBPF endpoint entry in cilium_ipcache map
4. Attaches TC programs to veth host end
5. Programs LXC (Linux Container) map entry for pod

IPAM Strategies#

Service Mesh Data Plane: Sidecar Injection#

Admission Webhook Injection#

Istio and Linkerd use Kubernetes MutatingAdmissionWebhook to inject sidecars:

1. Pod creation request → API server
2. API server calls webhook endpoint (istiod / linkerd-proxy-injector)
3. Webhook mutates pod spec: adds init container + sidecar container
4. Pod runs with sidecar

Init container (istio-init / linkerd-init): Sets up iptables rules in pod namespace. Sidecar container (envoy / linkerd2-proxy): Intercepts all traffic.

iptables Rules for Traffic Interception#

Istio init container configures the pod’s network namespace:

# Redirect inbound traffic to Envoy port 15006
iptables -t nat -N ISTIO_INBOUND
iptables -t nat -A PREROUTING -p tcp -j ISTIO_INBOUND
iptables -t nat -A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN  # Envoy health
iptables -t nat -A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN  # Envoy admin
iptables -t nat -A ISTIO_INBOUND -p tcp -j REDIRECT --to-port 15006

# Redirect outbound traffic to Envoy port 15001
iptables -t nat -N ISTIO_OUTPUT
iptables -t nat -A OUTPUT -p tcp -j ISTIO_OUTPUT
iptables -t nat -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN  # Envoy itself
iptables -t nat -A ISTIO_OUTPUT -p tcp -j REDIRECT --to-port 15001

UID 1337 is the Envoy proxy’s UID. The --uid-owner exemption prevents infinite recursion.

TPROXY vs REDIRECT#

Two iptables mechanisms for capturing traffic:

Istio uses REDIRECT (simpler). Linkerd uses REDIRECT by default. TPROXY is used when the proxy needs to know the original destination IP (e.g., L4 forwarding without DNS).

For Istio, Envoy reads the original destination via SO_ORIGINAL_DST socket option, which preserves the original destination even after REDIRECT.

Istio Ambient Mode: ztunnel Architecture#

Ambient mode (GA in Istio 1.22, May 2024) eliminates pod-level sidecars:

Components#

ztunnel (per node, DaemonSet):

• Handles L4: mTLS, authorization policy, telemetry
• One process per node, not per pod → lower overhead
• Uses WireGuard or HBONE (HTTP CONNECT over mTLS) for tunneling

Waypoint proxies (per namespace, optional):

• Envoy-based; handles L7 features when needed
• Only required for HTTP traffic management, JWT auth, L7 AuthorizationPolicy
• Scales independently from L4

Traffic Flow in Ambient Mode#

Pod A → ztunnel (node A) → [mTLS tunnel] → ztunnel (node B) → Pod B

If L7 needed:
Pod A → ztunnel (node A) → [mTLS] → Waypoint proxy → ztunnel (node B) → Pod B

Traffic capture mechanism: Ambient uses eBPF programs to redirect pod traffic to the ztunnel, without iptables rules in the pod namespace. No init container required.

xDS: Dynamic Proxy Configuration#

xDS (extension Discovery Service) is the API Istio uses to configure Envoy:

Envoy connects to istiod (pilot) and receives dynamic xDS updates via gRPC streaming. This enables zero-downtime configuration changes without restarting proxies.

Calico: BGP Routing Architecture#

Calico routes pod traffic using BGP (Border Gateway Protocol) by default:

• Each Kubernetes node runs a BGP speaker (via BIRD or GoBGP)
• Pod CIDRs are advertised via BGP to other nodes and physical routers
• No VXLAN overlay required in flat L2 networks
• BGP peering with ToR (Top-of-Rack) switches: enables pod IPs reachable from outside cluster

Node 1 (10.0.1.0/24) --BGP--> ToR Switch
Node 2 (10.0.2.0/24) --BGP--> ToR Switch
                               ↓
                         Pod IPs routable in datacenter

VXLAN mode: Available for environments without BGP (cloud VMs with no L2 control). Calico encapsulates packets in VXLAN, similar to Flannel.

eBPF dataplane: Calico optionally replaces iptables with eBPF (similar to Cilium). Requires configuring BPFEnabled: true — increases performance but same kernel requirements.

Linkerd: Rust Proxy vs Envoy#

Linkerd uses its own linkerd2-proxy written in Rust:

Linkerd’s proxy does exactly what Linkerd needs (mTLS, observability, retries, timeouts) with no extra capabilities. This makes it simpler to operate and lighter on resources.

Observability Integration#

Hubble (Cilium’s Observability Layer)#

Cilium’s Hubble provides per-flow observability using eBPF:

• hubble relay: Aggregates flow data across nodes
• Hubble UI: Web interface showing service dependency maps
• Hubble API: gRPC API for querying flows
• Prometheus metrics: L3/L4/L7 metrics per service pair

Unlike traditional monitoring, Hubble sees all traffic regardless of mTLS (eBPF operates below TLS). This enables observability even in encrypted service-mesh environments.

Distributed Tracing#

All major service meshes integrate with OpenTelemetry / Jaeger / Zipkin:

• Istio: Trace header propagation (x-b3-traceid, traceparent)
• Linkerd: Distributed tracing add-on via Jaeger extension
• Application must propagate trace headers (W3C Trace Context or B3 format)

S2 Recommendation: Container Networking & Service Mesh#

Technical Implementation Guidance#

New Kubernetes Cluster (Production)#

Recommended stack: Cilium (kube-proxy-free mode) + Hubble

# Helm values for Cilium
kubeProxyReplacement: true
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
encryption:
  enabled: true
  type: wireguard

Why: Single-vendor, eBPF throughout, observability included, no separate kube-proxy, WireGuard encryption available without service mesh complexity.

If Service Mesh Required#

For mTLS + observability only: Linkerd

linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -

Automatic mTLS for all annotated namespaces. Grafana dashboards out-of-the-box.

For full traffic management: Istio (ambient mode)

istioctl install --set profile=ambient
kubectl label namespace default istio.io/dataplane-mode=ambient

No sidecar injection. ztunnel DaemonSet handles L4. Add waypoint for L7 features.

Migrating from Weave Net#

1. Back up all workloads: kubectl get all --all-namespaces -o yaml > backup.yaml
2. Deploy Cilium in non-exclusive mode first (dual CNI migration)
3. Cordon and drain nodes one at a time
4. On each node: remove Weave Net, enable Cilium-only mode
5. Uncordon and verify pod connectivity

BGP Environments (Enterprise Datacenter)#

Use Calico with BGP configuration:

apiVersion: projectcalico.org/v3
kind: BGPPeer
spec:
  peerIP: 192.168.0.1   # ToR switch IP
  asNumber: 64512

Enables pod IPs to be routable across the datacenter without NAT.

Kernel Version Check#

Before deploying Cilium:

uname -r  # Need 5.10+; 5.15+ for full features

Ubuntu 22.04 LTS: 5.15 ✅ RHEL 9: 5.14 (mostly fine, minor feature gaps) RHEL 8: 4.18 ❌ (too old; upgrade required)

S2 Synthesis: Container Networking & Service Mesh#

Technical Landscape Summary#

Convergence on eBPF#

The most significant technical trend is eBPF replacing iptables across the networking stack:

• Cilium replaces both kube-proxy and CNI plugin with eBPF
• Istio ambient mode uses eBPF for traffic interception (eliminating init containers)
• Calico eBPF mode optionally replaces iptables
• Linux 5.10+ (widely deployed since 2021) enables these features everywhere

This is not just a performance optimization — eBPF enables new capabilities: per-packet observability (Hubble), WireGuard encryption with per-packet overhead, and atomic rule updates.

Service Mesh Complexity Barrier Falling#

Historically, the main objection to service meshes was operational complexity. Three developments address this:

1. Istio ambient mode (GA 2024): No per-pod sidecars; ztunnel per node reduces memory overhead from O(pods) to O(nodes)
2. Cilium Service Mesh: No sidecars at all; mesh functionality in kernel
3. Linkerd’s simplicity: Rust proxy is self-contained and operationally simple

The question is no longer “is a service mesh too complex?” but “which mesh fits our needs?”

The Weave Net Problem#

Weaveworks’ January 2024 shutdown leaves a significant installed base of clusters on abandoned CNI software. This is a security risk (unfixed CVEs) and an operational risk (no support). Clusters using Weave Net must migrate.

Calico’s Network Policy Advantage#

Kubernetes NetworkPolicy is limited: namespace-scoped, no cluster-wide policies, no egress to external IPs, no DNS-based policies. Calico’s GlobalNetworkPolicy and NetworkSet CRDs extend this significantly. For enterprises requiring fine-grained east-west and north-south policy, Calico remains the strongest option.

Linkerd vs Istio: Simplicity vs Features#

Linkerd is appropriate when mTLS + golden metrics are the primary needs. Istio is appropriate when traffic management (canary releases, fault injection, circuit breaking via DestinationRule) is needed.

Surprising Findings#

• Flannel lacks NetworkPolicy: Common misunderstanding. Flannel never implemented NetworkPolicy. Clusters that started with Flannel and added Calico’s policy controller are running two CNI plugins.
• Envoy is not a service mesh: It’s a proxy. Istio, Consul Connect, and others are meshes that configure Envoy. Understanding this distinction prevents confusion about what each component does.
• Istio ambient mode is GA: Some practitioners still think ambient is experimental. It graduated to Generally Available in Istio 1.22 (May 2024).
• Cilium can replace the whole stack: CNI + kube-proxy + service mesh in a single eBPF-based system. This reduces complexity significantly.

S3 Need-Driven Approach: Container Networking & Service Mesh#

Objective#

Map container networking and service mesh choices to concrete operational needs and developer personas. Identify when each tool is appropriate vs over-engineered.

Research Questions#

1. What are the use cases that require a service mesh vs. those that don’t?
2. How does a developer start with a local Kubernetes cluster and progress to production?
3. What are the migration paths from legacy setups?
4. What are the common patterns for transparent proxying in sidecar implementations?

Method#

• Documented use cases from simple single-node to enterprise multi-cluster
• Identified the decision triggers that upgrade from one tier to the next
• Compared self-built transparent proxy implementations vs. managed service mesh

Files in This Phase#

• use-cases.md — Use cases across maturity levels
• library-comparison.md — Feature comparison matrix for CNI and mesh tools
• recommendation.md — Persona-based recommendations

S3 Library Comparison: Container Networking & Service Mesh#

CNI Plugin Comparison#

Service Mesh Comparison#

Feature Matrix: Choosing a Service Mesh#

CNI + Mesh Combinations#

Proxy Memory Comparison#

For a cluster with 200 pods using service mesh:

Ambient mode and Cilium SM fundamentally change the economics of service meshes at scale.

Deprecated / Abandoned#

S3 Recommendation: Container Networking & Service Mesh#

Decision Matrix by Persona#

Small Team (1-20 developers, <50 nodes)#

CNI: Cilium (default configuration) Service Mesh: None initially; add Linkerd when mTLS is needed

Start simple: one CNI plugin that handles everything. Avoid service mesh until you have a specific need (compliance, canary deployments, cross-service circuit breaking).

Platform Team (20-200 developers, 50-500 nodes)#

CNI: Cilium with Hubble (kube-proxy-free mode) Service Mesh: Linkerd for zero-trust mTLS + golden metrics

Cilium + Hubble replaces the need for separate network monitoring. Linkerd is simple enough for a small platform team to operate while providing automatic mTLS for compliance.

Enterprise (200+ developers, 500+ nodes, multi-team)#

CNI: Calico (BGP peering with datacenter) or Cilium (if eBPF preferred) Service Mesh: Istio ambient mode

Calico for BGP integration with existing network infrastructure. Istio for the full traffic management feature set needed at enterprise scale (canary, fault injection, per-service circuit breakers, complex routing rules).

Migrating from Weave Net (Urgent)#

1. Do NOT wait — Weave Net is receiving no security patches
2. Migrate to Cilium (recommended) or Calico
3. Plan for a brief network disruption during node-by-node migration
4. Test in staging first: cilium connectivity test

Implementing a Custom Transparent Proxy#

If building custom proxy infrastructure (not using Istio/Linkerd):

1. Use iptables REDIRECT in the pod’s network namespace for simplicity
2. Use TPROXY only if original destination IP must be preserved
3. Init container pattern is the standard approach for sidecar injection
4. Consider the Envoy data plane directly if proxy logic is complex — Envoy already handles connection management, retries, circuit breaking, and health checking

When NOT to Use a Service Mesh#

• Fewer than 5 services that communicate with each other
• Services already use application-level TLS
• Team lacks operational capacity to manage control plane upgrades
• Non-HTTP protocols (TCP databases, MQTT, AMQP) that service meshes can’t inspect

S3 Use Cases: Container Networking & Service Mesh#

Use Case 1: Local Development Cluster#

Scenario: Developer runs Kubernetes locally (kind, minikube, k3d) for testing.

Requirements: Pods communicate, basic DNS, no multi-tenancy.

Solution: Accept default CNI (usually Flannel or kindnet). No NetworkPolicy needed. No service mesh needed.

Code example — minimal kind config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"

Use Case 2: Production Cluster, Basic Networking#

Scenario: Single-cloud Kubernetes cluster with standard microservices workload. Teams want NetworkPolicy for pod isolation.

Requirements: IP address management, NetworkPolicy, reasonable performance.

Solution: Cilium (default mode)

Key actions:

• Deploy Cilium via Helm with default settings
• Kernel 5.10+ available on Ubuntu 22.04/Debian 12/Amazon Linux 2023
• Enable Hubble for observability

Sample NetworkPolicy (works with any CNI):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-api
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Use Case 3: Enterprise Cluster with BGP Routing#

Scenario: Enterprise datacenter where network team controls BGP routing. Need pod IPs routable within datacenter (not behind NAT).

Requirements: BGP peering, GlobalNetworkPolicy, egress to specific external IPs, compliance-driven network segmentation.

Solution: Calico with BGP configuration

# Calico BGPPeer for ToR switch
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: tor-switch
spec:
  peerIP: 10.0.0.1
  asNumber: 64512
---
# GlobalNetworkPolicy (cluster-wide, not namespace-scoped)
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: deny-external-egress
spec:
  selector: app == 'payment'
  egress:
    - action: Allow
      destination:
        nets:
          - 10.0.0.0/8   # Internal only
    - action: Deny

Use Case 4: Zero-Trust mTLS Between Services#

Scenario: Compliance requirement that all service-to-service communication must be encrypted and mutually authenticated. Cannot modify application code.

Requirements: Automatic mTLS for all workloads, certificate rotation, policy enforcement.

Solution: Linkerd (simple case) or Istio (if L7 control also needed)

Linkerd setup:

# Install Linkerd
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Annotate namespace for injection
kubectl annotate namespace production linkerd.io/inject=enabled

# Verify mTLS
linkerd viz edges deployment -n production

Once injected, all TCP connections between annotated pods are automatically mTLS-wrapped. No application code changes. Certificates are rotated automatically every 24 hours.

Use Case 5: Canary Deployment with Traffic Splitting#

Scenario: Deploy new version of API service. Route 10% of traffic to v2, 90% to v1. Monitor error rate; roll back if errors spike.

Requirements: HTTP traffic splitting by weight, observability per version.

Solution: Istio with VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-service
spec:
  hosts:
    - api-service
  http:
    - route:
        - destination:
            host: api-service
            subset: v1
          weight: 90
        - destination:
            host: api-service
            subset: v2
          weight: 10
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: api-service
spec:
  host: api-service
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

Kiali dashboard shows per-subset traffic rates and error percentages.

Use Case 6: Transparent Proxy Sidecar (Custom Implementation)#

Scenario: Build a custom sidecar proxy that intercepts all traffic from a pod for logging, rate limiting, or protocol transformation — without application changes.

Requirements: Intercept all TCP traffic from pods; forward to proxy; proxy forwards to original destination.

Solution: Custom init container + transparent proxy using TPROXY

# Init container script (runs in pod's network namespace)
# Route all outbound traffic to proxy port 8888
iptables -t mangle -N PROXY_MANGLE
iptables -t mangle -A OUTPUT -p tcp -m owner --uid-owner 1000 -j RETURN  # skip proxy itself
iptables -t mangle -A OUTPUT -p tcp -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -j PROXY_MANGLE
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

# Then TPROXY in PREROUTING captures marked traffic
iptables -t mangle -A PREROUTING -p tcp -m mark --mark 1 -j TPROXY \
  --on-port 8888 --tproxy-mark 1/1

The proxy at port 8888 needs IP_TRANSPARENT socket option to receive packets with non-local destination IPs and read the original destination via SO_ORIGINAL_DST.

This is the core mechanism that Istio (via init container), Linkerd (via linkerd-init), and Cilium (via eBPF) use internally — just with different interception mechanisms.

Use Case 7: Multi-Cluster Service Mesh#

Scenario: Services span multiple Kubernetes clusters (multi-region or on-prem + cloud). Services in cluster-A need to call services in cluster-B transparently.

Requirements: Cross-cluster service discovery, mTLS across clusters, unified policy.

Solution: Istio multi-cluster (replicated control plane or single primary)

# East-west gateway in each cluster
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cross-network-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        number: 15443
        name: tls
        protocol: TLS
      tls:
        mode: AUTO_PASSTHROUGH
      hosts:
        - "*.local"

ServiceEntry objects expose services from cluster-B as virtual services in cluster-A. Cilium also supports multi-cluster via ClusterMesh (shared etcd for endpoint discovery).

S4 Strategic Analysis: Container Networking & Service Mesh#

The eBPF Lock-In Question#

Cilium’s eBPF approach is technically superior but raises a concern: is eBPF a stable platform to build on, or will it change in ways that break Cilium?

Assessment: eBPF is stable infrastructure.

• eBPF is part of the Linux kernel’s stable ABI guarantee
• The BPF verifier interface is backwards-compatible
• eBPF is now used by Meta (Facebook’s production infrastructure), Google (GKE), and cloud providers — the stability incentive is enormous
• eBPF Foundation (2021) provides cross-vendor governance

Risk: Kernel version fragmentation. Enterprises running older kernels (RHEL 8 with 4.18) cannot use Cilium’s full feature set. This is a real operational constraint for organizations with slow kernel upgrade cycles.

Vendor Concentration Risk#

Cilium#

• Primary maintainer: Isovalent (acquired by Cisco in December 2023)
• CNCF Graduated: Independent governance
• Contributor diversity: Microsoft, AWS, Google, Red Hat all contribute
• Risk level: Low — CNCF governance prevents Cisco from controlling the project; strong contributor diversity

Calico#

• Maintainer: Tigera (commercial company, not acquired)
• Not CNCF: Tigera controls the project
• Risk: Higher than Cilium — Tigera acquisition or bankruptcy could impact Calico
• Enterprise Calico: Tigera’s commercial product (Calico Enterprise) adds more risk to the free/OSS tier if Tigera pivots

Istio#

• CNCF Graduated (joined 2022, graduated 2023)
• Prior risk: Google controlled Istio before CNCF donation; now vendor-neutral
• Primary contributors: Google, Red Hat, Solo.io, IBM
• Risk level: Low — CNCF graduation and diverse contributor base

Linkerd#

• Original maintainer: Buoyant Inc.
• CNCF Graduated
• Controversy (2022): Buoyant attempted to move stable Linkerd releases behind commercial license. CNCF mediated; stable releases remain open source under Apache 2.0.
• Risk level: Medium — Buoyant controls most development; smaller contributor base than Istio; past licensing controversy shows commercial pressure

Weave Net#

• Dead: Weaveworks shutdown January 2024
• This is the cautionary tale: single-company OSS project with no CNCF governance → when the company fails, the project fails

Cloud Provider Managed CNI#

Major cloud providers offer managed CNI plugins:

GKE Dataplane V2 is built on Cilium — Google’s bet on eBPF as the networking future. Amazon VPC CNI takes a different approach: pods get actual VPC IP addresses (no overlay). This simplifies security group rules but limits pod density per node.

Strategic implication: Cloud providers are converging on eBPF (Cilium or Cilium-based). Choosing Cilium in self-managed clusters aligns with where managed Kubernetes is heading.

Istio Ambient Mode: Long-Term Trajectory#

Ambient mode represents a fundamental architecture change. Long-term implications:

Benefits that persist:

• O(nodes) memory for mesh functionality vs O(pods) — economically important at scale
• Simpler lifecycle (upgrade ztunnel DaemonSet, not every pod sidecar)
• No init container privileges required — better security posture

Current limitations (2026):

• Waypoint proxies required for L7 features add a new component to manage
• Some advanced Istio features not yet supported in ambient mode
• Operational tooling (debuggers, troubleshooters) still maturing

Direction: Ambient mode will likely become the default Istio installation profile within 2-3 years. New clusters should start with ambient.

Service Mesh Market Consolidation#

The service mesh space has been consolidating. Several projects have been abandoned:

• Linkerd 1.x: Replaced by Linkerd 2.x (different architecture)
• Conduit: Merged into Linkerd 2.x
• Maesh: Renamed to Traefik Mesh; limited adoption
• Nelson: Defunct
• AWS App Mesh: Deprecated December 2023 in favor of Amazon VPC Lattice

AWS App Mesh deprecation is significant: Amazon deprecated their service mesh product and directed customers to their layer-7 networking product (VPC Lattice) instead. This suggests that for AWS-native workloads, a managed L7 networking layer may replace the traditional service mesh pattern.

The “No Service Mesh” Alternative#

Service meshes add operational complexity. For clusters that primarily need:

• mTLS: Can be handled by application-level TLS (e.g., gRPC + cert-manager)
• Observability: OpenTelemetry SDK in applications + Prometheus
• Load balancing: Kubernetes Service + weighted ingress rules

The service mesh is often chosen for operational convenience (“automatic mTLS without code changes”) rather than because there’s no alternative. Teams should honestly assess whether the operational overhead is justified.

Trend: Cilium Service Mesh (sidecarless, eBPF) lowers the operational bar enough that the calculus is shifting. Cilium can provide mTLS, observability, and basic traffic management with less complexity than either Istio or Linkerd for many workloads.

S4 Strategic Approach: Container Networking & Service Mesh#

Objective#

Assess the long-term trajectory of container networking and service mesh tools. Identify risks from ecosystem consolidation, vendor dependencies, and technology shifts.

Research Questions#

1. Is the convergence on Cilium/eBPF durable or a temporary trend?
2. What happens if Cilium’s CNCF governance changes?
3. Is Istio’s complexity reduction (ambient mode) sustainable or still too complex?
4. What is the relationship between CNI plugins and cloud provider managed networking?
5. Will eBPF eventually standardize networking to the point CNI plugins become irrelevant?

Method#

• Analyzed CNCF governance structures for key projects
• Reviewed cloud provider managed Kubernetes CNI strategies
• Assessed maintenance signals (commit frequency, contributor diversity)
• Evaluated kernel/OS trends affecting eBPF viability

Files in This Phase#

• analysis.md — Strategic landscape, ecosystem risks, vendor relationships
• viability.md — 5-year viability scores and risk matrix
• recommendation.md — Long-term technology investment guidance

S4 Strategic Recommendation: Container Networking & Service Mesh#

Long-Term Technology Investment#

CNI: Invest in Cilium#

Cilium is the safe long-term bet:

• CNCF graduated with broad contributor diversity (Cisco/Isovalent, Google, Microsoft, AWS)
• Cloud providers converging on eBPF (GKE Dataplane V2 is Cilium-based)
• Replaces multiple tools: CNI plugin + kube-proxy + service mesh in one deployment
• eBPF stability backed by Linux kernel ABI guarantee

One caveat: If your organization runs kernels < 5.10, plan the kernel upgrade alongside Cilium adoption, or use Calico with VXLAN mode as an interim solution.

Service Mesh: Istio (Ambient) or Nothing#

If you need a service mesh for compliance (mTLS) or traffic management:

• Invest in Istio ambient mode — it’s now GA, and the trajectory is toward this being the default
• Avoid new sidecar-mode Istio deployments — the per-pod overhead is a known problem that ambient mode solves

If you only need mTLS + observability without complex traffic management:

• Linkerd or Cilium Service Mesh — simpler, lower operational overhead
• Evaluate whether application-level TLS + OpenTelemetry SDK isn’t simpler still

Avoid New Dependencies On#

• Weave Net (dead)
• AWS App Mesh (deprecated December 2023)
• Consul Connect without an existing HashiCorp Consul deployment (HashiCorp/IBM direction post-acquisition is uncertain)

Implementation Roadmap#

Quarter 1: Foundation#

• Standardize on Cilium as CNI (if not already using it)
• Enable Hubble for observability
• Replace kube-proxy with Cilium kube-proxy-free mode

Quarter 2: Security#

• Evaluate mTLS requirements (compliance, zero-trust mandate)
• If needed: deploy Linkerd or Cilium SM
• cert-manager for certificate management independent of mesh

Quarter 3: Traffic Management#

• If canary deployments or circuit breaking needed: add Istio (ambient mode)
• Deploy Kiali for traffic visualization

Quarter 4: Multi-Cluster#

• Cilium ClusterMesh or Istio multi-cluster for cross-cluster service discovery
• Evaluate Cilium’s BGP control plane as alternative to Calico for BGP use cases

Key Decision: Single Vendor vs Best-of-Breed#

Cilium-only (single vendor):

• CNI + kube-proxy replacement + service mesh + observability
• Fewer components, simpler upgrades
• Less L7 feature depth than Istio

Calico + Istio (best-of-breed):

• Best CNI for BGP environments
• Best service mesh for full traffic management
• Two separate upgrade tracks, two teams’ expertise needed

For new clusters without BGP requirements: Cilium-only is operationally simpler. For established enterprise clusters with BGP peering: Calico + Istio remains the standard.

S4 Viability Assessment: Container Networking & Service Mesh#

5-Year Viability Scores#

Risk Matrix#

Migration Effort Assessment#

Technology Bets#