We Killed the Management Cluster. The Tenant Clusters Didn't Notice.

The way we've always done it

That arithmetic is tolerable when you run one cluster. It becomes absurd when you run a hundred.

A hundred clusters means three hundred (300) control plane machines. Three hundred operating systems to patch. A hundred separate etcd clusters to back up, defragment, monitor, and occasionally restore at 3 AM. A hundred sets of certificates, each with its own expiry date, each one a future outage waiting for the day someone forgets. And every single one of those clusters takes hours to provision, because you're waiting on hardware or VMs before you can even begin bootstrapping.

For Cloud Providers, Managed Service Providers, and any organisation consuming Kubernetes at scale, this isn't an inconvenience. It's the business model's cost structure.

Enter the Hosted Control Plane

The Hosted Control Plane architecture asks a deceptively simple question: the Kubernetes control plane is just a set of stateless processes talking to a datastore. Why does it need machines of its own?

It doesn't. Run those processes as Pods, inside a Kubernetes cluster, and the picture changes entirely:

• Cost collapses at the hardware layer.

  Control planes become bin-packed workloads sharing a pool of compute, rather than reserved capacity sitting idle. That's the difference between three hundred servers and a management cluster you can count on your fingers.

• Provisioning goes from hours to minutes.

  Creating a cluster becomes creating a Deployment. There is no hardware in the critical path.

• High availability comes for free.

  A control plane that's a Deployment with three replicas, spread across failure domains with topology constraints, gets rescheduled automatically when a node dies. You inherit the management cluster's own resilience machinery instead of rebuilding it a hundred times.

• Day-2 operations become fleet operations.

  Upgrades, certificate rotation, and scaling stop being per-cluster runbooks and become reconciliation loops. This is exactly what our Kamaji codifies: it holds the Day-2 knowledge so a human doesn't have to.

• The datastore stops being a per-cluster problem.

  Instead of a hundred etcd clusters, you run a small number of pooled, properly-operated datastores and shard tenants across them, whether that's etcd or, via kine, a PostgreSQL or MySQL-compatible backend.

This isn't a fringe idea. It's how the hyperscalers have always run managed Kubernetes: your EKS, GKE, or AKS control plane is not three machines you can see, it's a workload on infrastructure you'll never touch, multiplexed with other people's. Gardener has run the seed/shoot model in production since 2018. Red Hat productised it as HyperShift. Kamaji is the vanilla, unopinionated implementation of the same pattern, using unmodified upstream components and kubeadm, producing fully CNCF-conformant clusters.

And then someone raises their hand

Every single time. At conferences, in architecture reviews, in the Q&A after talks I've given for years:

But if all the control planes run in one Management Cluster, isn't that a Single Point of Failure?

It's a good instinct. Concentration of anything deserves scrutiny. But the question conflates two very different properties: shared and single point of failure.

A component is a SPOF when its failure propagates to everything depending on it. The question is not "is this thing shared?", rather, it's "what actually happens to the dependents when it dies?"

So let's stop arguing about it and find out.

The experiment

At CLASTIX we're building kMetal, our enterprise offering built entirely on Kubernetes and Kamaji. The idea is to let Kubernetes tame the whole infrastructure rather than sit on top of it: we use Kubernetes to orchestrate bare metal instances directly, and to offer lightweight virtualisation via KVM where you need density.

The payoff is structural. With Kubernetes running on the bare metal itself, a tenant asking for a bare metal cluster doesn't trigger a hunt for three more servers to run a control plane on. The control plane is a workload on infrastructure that already exists. The same holds for VM-backed tenants.

Now, the environment I ran this on. It's our Development Environment: a single control plane node and a handful of workers. No HA. No quorum. One machine holding the entire brain of the management cluster.

You might read that as a weakness in the demonstration. It's the opposite. If I ran this against a properly designed three-node HA management cluster, killing one node would prove nothing: that's just Kubernetes doing what Kubernetes does. By running it on the most fragile possible topology, the demonstration gets harder, not easier. There is nowhere for the management cluster to hide.

This environment hosts Kamaji, the kMetal controllers, and the API Servers of the Tenant Control Planes that Kamaji manages.

Breaking it, properly

Step one: stop the kubelet on the management cluster's control plane node.

Here's the part that trips people up, and it's worth being precise about, because the intuition is usually wrong.

Stopping the kubelet does not stop the containers. The kubelet is an agent, not a supervisor of last resort. The actual process lifecycle belongs to the container runtime (containerd, in our case) and containerd neither knows nor cares that the kubelet has gone away. Every container on that node keeps running exactly as before: the API Server, etcd, the scheduler, the controller-manager, all of it.

What does change is that the node stops sending heartbeats. The kubelet renews a Lease object in the kube-node-lease namespace roughly every ten seconds (although it can be customised), and when those renewals stop, the node-lifecycle-controller notices after --node-monitor-grace-period elapses (40 seconds by default historically, raised to 50 seconds in Kubernetes 1.32) and flips the node's Ready condition to Unknown. The node shows up as NotReady, gets tainted with node.kubernetes.io/unreachable:NoExecute, and the scheduler stops placing anything new on it.

Then, five minutes later: the DefaultTolerationSeconds admission controller injects a 300-second toleration into every Pod that doesn't specify its own, and eviction kicks in. The Pod objects get deleted from the API. Their controllers create replacements elsewhere.

But the containers on the dead node? Still running. There's no kubelet left to receive the deletion and actually tear anything down. This is the classic "zombie workload" scenario, and it's exactly why the demo can't stop here.

There's a delicious recursion in it, too: on a single-control-plane cluster, the controller-manager that marks the node NotReady is itself a static Pod on that node, still happily running under containerd, dutifully reporting that its own host has failed.

Step two: kill the API Server.

This is where killing the kubelet first turns out to be load-bearing. With a healthy kubelet, killing the API Server container achieves nothing: the kubelet notices the static Pod's container is gone and restarts it within seconds. You have to remove the resurrection mechanism before the kill will stick.

So: kubelet down, then API Server down. And on a single-control-plane cluster, that's total brain death. The scheduler and controller-manager are still processes in memory, but they have nothing to talk to. Nothing can be created, updated, deleted, or reconciled anywhere in the management cluster. etcd is still there, holding state that nobody is reading.

The management cluster is, for all practical purposes, gone.

And now, the interesting part

I pick up the kubeconfig for a Tenant Cluster managed by Kamaji, and I run kubectl get nodes.

It works.

Namespaces: fine. Pods: fine. And not just reads: I roll out a new Deployment in the tenant cluster and watch Pods get scheduled, pulled, started, and become Ready. The tenant cluster is not degraded, not read-only, not limping. It is completely unaware that anything happened.

Why this works

This is the heart of it, and it deserves more than a shrug.

Kubernetes' control plane is a control plane, not a data path

The single most important thing to internalise: nothing in a running Kubernetes cluster routes user traffic through the API Server. Not one packet.

When a request hits a Pod, it traverses iptables or IPVS rules that kube-proxy programmed into the node's kernel some time ago (or the eBPF maps when picking up a kube-less CNIs). It crosses a network that the CNI plugin configured when the Pod was created. It lands in a container that containerd is supervising. The API Server was involved in deciding that this arrangement should exist. It plays no part whatsoever in executing it.

Kubernetes is a reconciliation engine sitting beside the data plane, not a proxy sitting in front of it. Kill the reconciliation engine and the data plane keeps doing the last thing it was told to do: indefinitely, and at full speed.

Once you see that, everything else follows.

The Tenant Control Plane is just processes, and processes don't need permission to keep running

A Kamaji-managed Tenant Control Plane is a Deployment of ordinary Pods running upstream kube-apiserver, kube-scheduler, and kube-controller-manager. Those Pods are scheduled onto management cluster worker nodes, where containerd runs them.

The management cluster API Server's role in that Pod's life ended the moment it was scheduled. It is not a supervisor, not a proxy, not a dependency of the running process. It's the thing that decided the Pod should exist. Once it does exist, the API Server can vanish, and the process carries on, because it's just a process on a Linux box.

The tenant's state lives somewhere else entirely

The Tenant Control Plane is stateless by design: Kamaji pushes all state into a DataStore, at your wish an independent (or pooled) etcd cluster, or a kine-backed SQL database, which is its own independent system. A tenant's kube-apiserver reads and writes tenant state directly to that datastore. The management cluster's own etcd holds Kamaji's CRDs and the management cluster's Kubernetes objects. These are two completely separate data paths that happen to share a data centre.

Tenant worker nodes never talk to the management cluster

This is architectural, not incidental. Kamaji's model is deliberately unidirectional: the Management Cluster knows about Tenant Clusters, and Tenant Clusters have no awareness of the Management Cluster's existence: essentially, it’s the top-down approach from design patterns applied in software engineering, but applied to infrastructure.

A tenant's kubelet is configured with the endpoint of its API Server (a LoadBalancer VIP, a NodePort, whatever the TenantControlPlane exposes) and that's the only Kubernetes endpoint it has ever known. It connects to a socket. It has no idea, and no way of finding out, that the process behind that socket happens to be a Pod. There is no management cluster credential on a tenant node. There is nothing to break.

So the tenant cluster is fully functional, not merely alive

This is the point worth labouring. It isn't that tenant workloads survive in some frozen state. The tenant's own control plane is intact and doing its job: its scheduler schedules, its controller-manager reconciles, its API Server serves. Rolling updates work. Autoscaling works. Failed Pods restart. Tenant admins can do anything they could do yesterday.

We didn't take out the tenants' control planes. We took out the control plane of the control planes: one level up. Everything below that boundary is untouched.

The dotted lines are what broke. The solid lines are the ones that carried on.

What actually does break

I'd be doing the architecture a disservice if I stopped at the victory lap. A dead management cluster is a real incident with real consequences: they're just not the ones people assume.

The correct mental model is that a failed management cluster loses the ability to change things, not the ability to run things. Specifically:

No reconciliation means no self-healing. A running Tenant Control Plane Pod survives happily. A Tenant Control Plane Pod that crashes during the outage will not come back, because the thing that would recreate it is dead. Your exposure grows with the duration of the outage. This is precisely why running Tenant Control Planes with multiple replicas, spread across nodes with anti-affinity and topology spread constraints, isn't optional in production.

Stale EndpointSlices can blackhole traffic. Tenant kubelets typically reach their API Server through a management cluster Service. Existing kube-proxy rules stay programmed and keep working, but if the set of backing Pods changes while the control plane is down, nothing updates the EndpointSlice, and traffic can be directed at endpoints that no longer exist. Layer this correctly: a stable VIP, health-checked load balancing, and don't make your tenants' API Server reachability depend on a resource that only a live control plane can refresh. This effect is pretty similar to what happens to an Ingress (or Gateway API) Controller when the API Server is unresponsive: it still routes traffic, but rules aren’t updated.

So no, this demo is not a licence to run your management cluster carelessly. It should still be genuinely HA with a proper etcd quorum, spread across failure domains, backed up and tested for restore, protected by PodDisruptionBudgets, and critically monitored. All the discipline you'd apply to any production cluster still applies. Arguably more, because it's the highest-leverage cluster you own.

The comparison nobody makes

Here's what the SPOF objection quietly assumes: that the alternative has no single points of failure.

It has a hundred of them.

Every one of those hundred traditional clusters has its own etcd quorum, and losing two of three members takes that cluster down hard. Every one has its own certificate expiry, its own upgrade risk, its own three machines that someone has to patch, and their own hypervisors if running as VMs. You haven't eliminated the failure domain: you've replicated it a hundred times and handed it to a team that can't possibly give each instance the attention it needs.

The honest comparison isn't one shared risk versus no risk. It's one well-engineered, well-monitored, well-funded failure domain versus a hundred under-maintained ones. And critically, as this experiment shows, the shared domain's blast radius is bounded to the control operations. The hundred individual ones each take their entire cluster with them.

Fleet-wide, the architecture that concentrates the risk into something you can actually afford to engineer properly is the one that fails less.

Where this leaves us

We took the least resilient management cluster we had, stopped its kubelet, killed its API Server, and watched every Tenant Cluster carry on serving traffic, scheduling Pods, and answering kubectl. That's the Hosted Control Plane architecture and the hard multi-tenancy Kamaji implements, working exactly as designed.

The myth doesn't survive contact with the experiment.

This is why we built kMetal: Kubernetes shouldn't sit on top of your infrastructure, it should be your infrastructure. kMetal is a multi-tenant platform built entirely on the Hosted Control Plane model, which also gives you a rather elegant answer to Kubernetes Security Posture Management: no Control Planes, no access to the platform. alongside multi-tenant networking and multi-tenant storage.

Hard, secure clusters in minutes, not months.