In this post, we explore how Kairos, an immutable operating system tailored for Kubernetes, and Kamaji, a Kubernetes control plane manager for multi-tenant or hosted clusters, come together to deliver a powerful and secure architecture. We'll dive into how they work individually, how they integrate, and why their combination is a game-changer—especially in environments where the state must be preserved, like with Ceph storage nodes.

🔗 Watch the full demo on YouTube

The Case for Immutable Kubernetes Worker Nodes

Traditional Kubernetes deployments rely on mutable infrastructure, where changes are applied in place — patches, config updates, and upgrades can introduce drift, inconsistencies, and vulnerabilities. Immutable infrastructure flips that model: instead of changing a node, you rebuild it from a known-good, declarative image.

Why this should be considered:

• Enhanced Security: Immutable nodes drastically reduce the attack surface by only exposing what is strictly necessary. This is especially critical on the data plane, where workloads execute.

• Supply Chain Integrity & SBOM: every system build is tied to a reproducible Software Bill of Materials (SBOM), enabling traceability and auditability at the OS level.

• Predictable Upgrades: Since each artefact is pre-tested and versioned, updates are simply a matter of deploying a new node with a new image, eliminating drift and reducing the chance of runtime misconfigurations.

Separation of Concerns: Control Plane vs. Data Plane

This separation is already standard in managed Kubernetes services like GKE, AKS, and EKS. You don’t manage the control plane—just the worker nodes.

Kamaji brings this paradigm to your own infrastructure. It allows you to host multiple lightweight Kubernetes control planes as custom resources inside a "management cluster", reducing resource usage, operational complexity, and TCO.

🧠 Kamaji implements the Hosted Control Plane pattern, where the control plane is decoupled from the data plane—allowing you to manage thousands of Kubernetes clusters efficiently.

We already discussed in one of our previous blog post by Adriano Pezzuto.

This decoupling becomes crucial when building platforms on bare metal, where rebuilding machines isn’t always feasible (or desirable).

Meet Kairos: Immutable OS for Edge and Cloud-Native Infrastructure

Kairos is a lightweight, immutable Linux OS that bootstraps itself by supercharging the cloud-init system, creating reproducible and verifiable infrastructure. Kairos nodes boot from pre-built images, can auto-join Kubernetes clusters, and are optimized for air-gapped, secure, and edge deployments.

Considering our use case, let's understand why this approach is a win scenario:

• You can perform in-place upgrades via an A/B partition scheme.

• Nodes are stateless unless explicitly configured to retain state.

• Comes with SBOM support out of the box, integrated with the build pipeline.

• Supports automatic rollbacks if something goes wrong during an upgrade.

This contrasts starkly with traditional Cluster API-based workflows, which assume you can replace nodes entirely—a problematic assumption in stateful environments like Ceph clusters or database nodes.

The Power of Combining Kamaji + Kairos

The integration between Kairos and Kamaji allows you to:

• Provision immutable worker nodes (Kairos) that connect to lightweight hosted control planes (Kamaji).

• Enable multi-tenancy at scale with reduced control plane overhead.

• Support stateful workloads by enabling in-place OS upgrades without triggering full node replacement.

• Enforce strong security boundaries by keeping the control plane out of reach of the worker node infrastructure.

I already highlighted how to combine these two approaches in this video demo, by watching it

• Kamaji running a hosted control plane inside a management cluster.

• Kairos being used to deploy a secure, minimal OS image onto an instance (AWS for the demo, but it can bare also a bare metal server).

• The Kairos node then bootstrapping itself and connecting to the Kamaji-hosted control plane seamlessly.

Use Cases

1. Bare Metal with Persistent State

For workloads like Ceph or distributed databases where nodes hold critical persistent data, replacing the machine isn't acceptable. In-place, immutable upgrades (A/B booting) are critical.

2. Multi-Tenant Edge Infrastructure

Run thousands of isolated Kubernetes clusters with minimal resource overhead using Kamaji, while booting all worker nodes via Kairos from signed, trusted images.

3. Air-Gapped or High-Security Deployments

Immutable infrastructure + verifiable supply chain = a secure, repeatable, and audit-friendly system.

Conclusion

Combining Kamaji and Kairos brings cloud-native operational patterns—immutability, hosted control planes, and zero-trust architecture—to bare metal and edge environments. Whether you're building the next platform for edge clusters, managing secure IoT fleets, or running data-intensive systems on metal, this architecture is worth serious consideration.

🔗 Watch the full integration demo on YouTube

Ready to Dive Deeper?

If you're interested in implementing this architecture or exploring how it can fit into your infrastructure strategy, we invite you to connect with us. As pioneers of the Hosted Control Plane model and core maintainers of Kamaji, CLASTIX is already collaborating closely with the Kairos community to enable production-grade, scalable, and secure Kubernetes deployments.