Kill the Control Plane: What Istio, Linkerd and Consul Do Next

I scaled istiod to zero and the traffic did not notice. Thirty-seven requests over nine minutes, every one a 200, including the stretch after the workload certificates had already expired. Linkerd and Consul did the same thing. If your control plane failure drill is “kill it and send a request”, all three meshes pass and you have learned nothing.

What survives is the connection you already had. All three keep an established connection alive and encrypted with no control plane at all. The differences start the moment anything has to be new: a new connection, a new pod, a new policy. That is where these three stop agreeing, and where one of them fails open.

Same cluster template as Part 1, same two-service app, same marker string in the response body so I can tcpdump the wire and grep for it. For each mesh I scaled the whole control plane to zero: istiod for Istio, all three of linkerd-destination, linkerd-identity and linkerd-proxy-injector for Linkerd, and the server plus the connect injector for Consul.

Existing traffic survives, in all three

I got this wrong the first time, and the mistake is worth more than the result. My first Linkerd run said existing traffic died within seconds of the control plane going away. It had not. Every sample in that run was a fresh kubectl exec curl, so I was measuring the cold path and calling it existing traffic.

What makes that trap easy to fall into is that the identical method gives you opposite answers depending on the mesh. Istio’s Envoy holds idle upstream connections in a pool, so my separate clients all rode one proxy-to-proxy connection that was already open and the test looked like a pass. Linkerd’s proxy does not keep that connection sitting idle, so the same test opened a new one every time and looked like a catastrophic failure. Neither reading was about resilience. Both were about connection pooling.

Held properly open, with one curl process reusing a single connection across the kill, Linkerd returned 45 of 45. If your resilience drill reconnects between samples, it is not testing what you think it is.

A new pod is where they split

Proxy injection is a mutating admission webhook, and failurePolicy decides what Kubernetes does when that webhook is unreachable. That one field decides most of what follows.

Consul’s policy is the one you cannot read from its chart, because the pod webhook is created at runtime by the injector. On a live cluster it is Fail.

Linkerd is the outlier, and “admitted with no proxy” undersells it. The pod is created, Kubernetes marks it Ready, it has no linkerd-proxy container and no init container, and nothing anywhere reports a problem. Then it talks to the meshed backend:

It joined a meshed namespace as an unmeshed workload. No mTLS, and outside every Server and AuthorizationPolicy that would otherwise cover it.

Then I reproduced it by accident, which is the part that convinced me. While restoring the cluster I ran rollout restart a few seconds before the injector was ready. Both replacement pods came back 1/1 instead of 2/2, silently unmeshed, no error. That is how this actually bites: not somebody running kubectl run during an incident, but ordinary automation restarting a workload at the wrong moment.

Policy changes are refused, or accepted and armed

Linkerd’s validating webhooks are Ignore too, so a policy applied blind during an incident lands in the API, does nothing while the proxies cannot learn it, and then takes effect the moment the control plane returns. I confounded my own recovery measurement with exactly this: a deny-by-default Server I had applied during the outage fired on recovery and I briefly read it as a failure to recover.

Istio’s refusal is more annoying and much safer. You cannot change anything, and you know it immediately.

The health tools do not agree on honesty

Istio is honest and immediate. Note what it still cannot tell you: nothing in istioctl reports on the data plane that is currently carrying your traffic, because it asks istiod, and istiod is gone.

Linkerd is the one to watch. With all three deployments scaled to zero it reported this:

linkerd-existence
-----------------
√ control plane replica sets are ready
√ no unschedulable pods

Both ticks are true and both are vacuous. There are no unschedulable pods because there are no pods. Then the command stops returning; I killed it after three minutes.

Consul fails in a category of its own. Its CLI lives inside the server pod, so the command you would use to diagnose the outage is deleted by the outage: Error from server (NotFound): pods "consul-server-0" not found. The diagnostic tool has the same blast radius as the thing it diagnoses.

Recovery is automatic in two of them

Istio and Consul both healed themselves with no intervention:

Both reissued fresh certificates to every proxy, and Consul’s pods were still 2/2 with zero restarts seventy minutes in.

Linkerd did not heal at all. Four minutes after the control plane was healthy again, the workload proxies were still failing, still holding their expired certificates, retrying every ten seconds:

ERROR linkerd_proxy_identity_client::certify: Failed to obtain identity
  error=... BrokenPipe, "stream closed because of a broken pipe"

Meanwhile the identity controller was healthy and issuing certificates normally, to the control plane’s own components. It logged no attempt at all from my workloads. A pod with no certificate bootstraps fine; a pod with an expired one did not. Restarting the client alone was not enough either, because the backend was stuck the same way and answered 504. Only after restarting every meshed workload did traffic return.

So for Linkerd, recovery from an outage that outlives the certificate lifetime means restarting your whole mesh, and restarting is also the operation that silently drops the proxy if the injector is not back yet.

The certificate clock, last

The certificate deadline is the number everyone quotes, and it bites last rather than first. It is also the one place a mesh could plausibly do something quietly unsafe, which is why the marker string exists: if a proxy ever gives up on mTLS and falls back to cleartext, the marker shows up on the wire.

No capture ever showed it, under any of the three.

Istio is the cleanest demonstration. Two clients, identical except that one had been talking to the backend all along and the other had been kept deliberately silent, so its first request had to complete a real handshake between two expired peers:

The warm client never opened a connection, so no handshake happened and the expiry was never consulted. That is the entire explanation for traffic outliving its own certificate. Envoy states the cold client’s reason itself:

503 URX,UF upstream_reset_before_response_started{remote_connection_failure|
  TLS_error: ... CERTIFICATE_VERIFY_FAILED ... certificate_has_expired}

It fails closed, and the error names the real reason rather than a generic connection failure.

Consul does the same thing on the same schedule. Its certificates ran 05:07:58Z to 06:07:58Z, and a connection held open across that boundary returned 95 of 95, while a probe opening a fresh connection every 55 seconds returned 200 fifty-eight times and then failed on the first sample after expiry, at 06:08:32Z. A capture past expiry found 76 packets, no marker and no plaintext.

The difference is what each one tells you afterwards. Istio names the cause. I grepped both Consul dataplanes for certificate, TLS_error, expired and handshake and got nothing at all: the request just stops. Same failure, same moment, and one of them leaves you to guess.

One practical note if you want to reproduce this: all three let you shorten the certificate lifetime, but not equally. Istio’s SECRET_TTL is not on the doc pages I checked and accepted 10 minutes (it issued 12). Linkerd’s identity.issuer.issuanceLifetime accepted 10 minutes and issued 10m40s, which is exactly the lifetime plus its 20 second clock skew allowance at each end. Consul’s leaf_cert_ttl has a hard floor of one hour, and setting it lower does not fail your install. The server sits there 1/1 Running with the CA uninitialized and nothing can mesh at all.

What the three parts add up to

Part 1 pointed an unauthorized pod at all three meshes and found the security boundary is smaller than it reads, because every one of them served that pod real data through the endpoint it had built for the kubelet’s probe. Part 2 moved the mechanism out of the pod with Istio’s ambient mode, which closed that hole and blinded Kubernetes health checks in the same move. This part asks what is left when the mechanism is gone.

The answer is consistent across all three: a service mesh’s control plane is not in your request path, and its absence is invisible until something reconnects. Everything already running keeps running, encrypted, indefinitely. Every mesh then fails on the next new thing, and what differs is whether it fails closed and says so.

Istio and Consul fail closed on new pods and new config. Linkerd, alone, admits an unmeshed pod into a meshed namespace and calls it Ready.

So drill the reconnect, not the request. Kill your control plane, then start something: scale a deployment, evict a pod, restart a node. If your mesh is Linkerd, check that the replacement pod came back 2/2, because nothing else will tell you it did not.