How Service Meshes Actually Work: Sidecars, mTLS, and Traffic Shaping

Quick Answer

A service mesh is a fleet of per-service sidecar proxies (Envoy for Istio, the Rust-based linkerd2-proxy for Linkerd) plus a control plane that configures them all consistently. Every request between services gets silently intercepted by these proxies, which handle mutual TLS, retries, weighted traffic splitting for canary releases, and uniform metrics — all without a line of application code changing.

What a mesh adds on top of a sidecar

If you haven't read Priodemy's sidecar pattern post, the short version you need here is: a sidecar is a second process deployed alongside your application container that transparently intercepts its network traffic, usually via iptables redirection, so the application keeps making normal-looking network calls without knowing anything is in the way. That's the mechanism. This post is about what gets built on top of that mechanism at scale.

A service mesh is what you get when every single service in your system has one of these sidecars, and a control plane configures all of them from one place. In Istio, that control plane component is istiod; it pushes configuration and certificates out to every Envoy sidecar in the mesh. In Linkerd, the control plane's identity component plays the equivalent role for certificates specifically. The point of centralizing this is consistency — you set an mTLS policy, a retry budget, or a traffic-split rule once, and every proxy in the mesh enforces it the same way, regardless of what language or framework the service behind it is written in.

mTLS: how two sidecars actually authenticate each other

"Encrypted traffic between services" undersells what's happening. In Istio, when service A calls service B, the request is first rerouted to A's local Envoy sidecar. According to Istio's own security documentation, that client-side Envoy then "starts a mutual TLS handshake with the server side Envoy" — and during that handshake, it also performs what Istio calls a secure naming check: it verifies that the service account presented in B's certificate is actually authorized to run the destination service being called, not just that the certificate is validly signed. Only after that check and the TLS handshake succeed does A's Envoy forward the request to B's Envoy, which independently authorizes it before passing it over a local, plaintext connection to B's actual application container. Two separate Envoys did two separate verification steps before your application code ever saw the request. Istio controls this with two resource types: PeerAuthentication configures what a sidecar will accept, and DestinationRule configures what it will send.

Linkerd's identity model is worth contrasting because it's structurally different. Per Linkerd's documentation, the control plane runs a certificate authority component called identity. Each data-plane proxy generates its own private key on startup, inside a tmpfs directory that lives only in memory and "never leaves the pod" or touches disk. The proxy then sends a certificate signing request to the control plane, using its pod's Kubernetes ServiceAccount token as proof of who it is, and gets back a signed certificate scoped to just 24 hours before it has to repeat the whole process. So every workload in a Linkerd mesh has a cryptographic identity tied directly to its ServiceAccount, freshly issued and automatically rotated, with the private key never persisted anywhere outside that one pod's memory.

Traffic shaping: canary releases without touching deployment logic

Because every request already passes through a sidecar that the control plane configures, that same sidecar can make routing decisions — not just security ones. Istio's traffic management model splits this into two resources: a VirtualService decides where traffic for a given hostname actually goes, and a DestinationRule defines policy (load balancing, connection pooling, outlier detection) for what it's routed to, applied after the VirtualService's routing decision.

The canary use case follows directly from this: you can configure a percentage-weighted split — Istio's own documentation uses the example of 75% of traffic to version v1 and 25% to v2 — and gradually shift that weight as you gain confidence in the new version. The important architectural detail here, stated directly in Istio's docs, is that this routing is completely decoupled from how many instances of each version you're running: you can scale v2 up or down for capacity reasons without that changing the 75/25 traffic split at all, because the split is a routing rule enforced at the proxy layer, not a side effect of instance counts the way naive load balancing would be.

Observability you get without instrumenting anything

Because literally every service-to-service call passes through a sidecar, that sidecar is a natural place to record uniform telemetry — request latency, error rates by status code, retry counts, and connection-level stats — for every service in the mesh, in the same format, regardless of what that service is written in. A Python service and a Go service behind Envoy sidecars produce directly comparable metrics without either one needing a metrics library. This is one of the more underrated reasons teams adopt a mesh: it turns "we need every team to instrument their service consistently" from an org-wide discipline problem into an infrastructure default.

The trade-off is that this observability is entirely at the network layer — it tells you a request from A to B took 340ms and got a 200, not why your code took 340ms internally. Distributed tracing headers get propagated through the mesh, but your application still has to participate in the trace for it to be useful inside a single service's boundary.

The cost: latency, memory, and a new thing to operate

None of this is free. A single service-to-service call that used to be one hop is now at minimum four hops: your app to its local sidecar, that sidecar to the destination's sidecar, that sidecar to the destination app, and the response retracing the same path. Each hop adds latency, and the mTLS handshake specifically adds CPU cost for the cryptographic operations on every new connection (mitigated in practice by connection reuse, but still a real cost the first time). Every pod in the mesh also now runs an extra container, which means extra memory and CPU reserved per pod across your entire fleet, not just on a few gateway nodes.

Then there's the control plane itself: istiod or Linkerd's control plane components are now critical infrastructure you have to operate, monitor, and upgrade, with their own failure modes — a broken or unreachable identity/certificate-issuing component can, in the worst case, stop new connections across your whole mesh from establishing once existing certificates expire.

Istio vs Linkerd: two different bets on the same idea

Istio is built around Envoy, a general-purpose proxy written in C++ that was originally built at Lyft and is also used standalone as an API gateway and ingress controller outside any mesh context. That generality is Istio's main selling point and its main complexity cost: Envoy's configuration surface is enormous, and Istio layers its own CRDs (VirtualService, DestinationRule, PeerAuthentication, and more) on top to make it usable.

Linkerd took the opposite bet: it built its own purpose-specific proxy, linkerd2-proxy, written in Rust specifically to be minimal, fast, and to do the one job of being a mesh sidecar well rather than being a general-purpose proxy that also happens to support mesh use cases. In practice this usually shows up as a smaller resource footprint per sidecar and a smaller configuration surface to reason about, at the cost of some of the flexibility Envoy's general-purpose design gives Istio.

When you actually need a mesh

A mesh earns its overhead when you have enough services, owned by enough different teams, that enforcing consistent mTLS, retries, and observability by convention has already started failing — someone forgot to add the retry logic, someone's TLS config drifted, someone's metrics format doesn't match anyone else's dashboards. Below that scale, the extra sidecar per pod, the extra control plane to run, and the extra failure surface usually cost more than the problem they'd be solving. A handful of services that one team owns and can keep consistent by hand rarely needs this; dozens of services across multiple teams, all needing the same security and observability guarantees without re-implementing them per service, is the actual target case.

Frequently Asked Questions