Cover image for LiveWyer blog post: Namespaces, Clusters, or Virtual Clusters? Choosing a Kubernetes Multi-Tenancy Model
Engineering • 10min read

Namespaces, Clusters, or Virtual Clusters? Choosing a Kubernetes Multi-Tenancy Model

Why namespace-per-team platforms drift into cluster sprawl, and where virtual clusters fit between namespaces and dedicated clusters.

Written by:

Avatar Louise Champ Louise Champ

Published on:

Last updated on:

Most platform teams start multi-tenancy with a namespace per team. It is cheap and means there is only one cluster to upgrade, while RBAC, NetworkPolicy, and ResourceQuota keep each team inside its own namespace.

What tends to happen next, however, is when a team needs to install an operator which has been written on the assumption that it would be the only tenant on the cluster. If its custom resources are cluster-scoped as opposed to namespaced, on top of its CustomResourceDefinitions (CRDs) and cluster-scoped RBAC, which every tenant shares, the custom resources themselves are now shared with every other tenant.

Installing it on the shared cluster changes the API for everyone, so you decide that team gets a cluster of its own. The next team with the same need gets one too, and the estate ends up as a mix of shared multi-tenant clusters and single-tenant clusters, with a platform team running both.

Having been in this scenario previously, with hindsight, we would have used virtual clusters instead. The reason comes down to three dials which pull against each other in any multi-tenant platform:

  • how well tenants are isolated from each other,
  • how much control each tenant has over its environment,
  • and how much the whole arrangement costs to run.

Each tenancy model turns one of those dials up by turning another one down, so the choice you land on may depend on where you need each one set. Let’s go through each dial in a bit more detail.

Dial 1: isolation, mostly from accidents

With internal teams, the isolation you need is mostly protection from accidents, as opposed to protection from attackers.

The incidents you tend to see on shared clusters are mundane, such as a batch job with no limits that starves a node, or a namespace where nobody created a NetworkPolicy which implements a default-deny. Namespaces handle these well once a ResourceQuota and LimitRange are in place, network policies deny traffic by default, and Pod Security Admission runs at the restricted level.

However, a namespace is ultimately just a boundary in the Kubernetes API. Underneath it, all the tenants still share nodes, kernels, and the control plane. For most internal teams that is an acceptable risk, and the exceptions are services which are too critical to share an outage with, and code you do not trust.

If a team that runs login, or the API every other service calls, shares a cluster, a bad upgrade of the host cluster or an outage in one of the platform’s own webhooks takes it down along with everyone else. A virtual cluster does not change that just because it runs on the same host.

CI runners that build pull requests from outside the organisation are the usual case of untrusted code, and those workloads would need a sandboxed runtime (such as gVisor or Kata Containers), dedicated nodes, or a cluster of their own.

Dial 2: control, which namespaces cannot give

In a cluster, anything cluster-scoped is shared by every tenant on that cluster.

A cluster runs one copy of each CRD. If two teams try to run different versions of the same operator, both operators will try to own that one CRD, so an upgrade by either team changes the schema and served versions for the other, and an operator that reapplies its CRD on start-up will change it back.

This scenario is worse when the custom resources themselves are cluster-scoped, as they were in our example: there is no namespace to keep one team’s objects apart from another’s, so RBAC cannot scope access by team and the operator acts on every team’s objects at once.

The same applies to ClusterRole and ClusterRoleBinding, StorageClass, IngressClass, PriorityClass, and admission webhooks. A mutating webhook with a failurePolicy of Fail that goes down will block pod creation in every namespace it targets, not just the namespace of the team that installed it.

So on a namespace-based platform, the platform team becomes the gatekeeper for all cluster-scoped changes. When requests are rare, that can work, but stops working once a team’s product depends on an operator they need to upgrade on their own schedule. It is at this point where the team in our example was moved to its own cluster.

Tooling such as Capsule can reduce the ticket load. It groups namespaces into a tenant, lets a tenant owner create namespaces without asking the platform team, and applies policy across all of them. That takes a lot of repetitive requests away, but the CRDs and webhooks are still shared.

Dial 3: cost, which grows with each cluster

Adding an extra cluster ends up costing more in engineering time and idle nodes than it does in just the usual control plane fees.

The control plane itself is the cheap part. EKS charges $0.10 an hour per cluster in standard support, which is about $73 a month. What ends up costing more is the extra resources this new cluster requires:

  • An additional deployment of the add-on stack, with ingress, cert-manager, monitoring and logging agents, a policy engine, and secrets integration each becoming one more version to track.
  • Headroom which cannot be shared. Spare capacity on one team’s nodes cannot absorb a spike on another team’s, so each cluster carries idle nodes the others cannot use.
  • Its own upgrade cycle. Kubernetes releases about three times a year, and skipping a minor version is unsupported, so the clusters all have to step through each release and have their add-ons checked against it.

The mixed estate is the hardest model to run

Set against these three dials, we can score the models we see on internal platforms like this:

ModelIsolationTenant controlCost efficiencyPlatform team effort
Namespace per teamPartialWeakStrongLow
Cluster per teamStrongStrongWeakHigh, and grows with each cluster
Mixed estateVaries by teamVaries by teamPartialHighest, two operating models
Virtual cluster per team (shared nodes)PartialStrongStrongMedium

A namespace per team is the cheapest model to run, and it is the right default for teams that deploy applications and do not need anything cluster-scoped. A cluster per team gives each team full control and a hard boundary, but the cost of that falls on the platform team with every cluster that is added.

The mixed estate is where the platform in our example ended up. Each team was provided with a space in the platform which was appropriate to them and their use-cases, but the platform team now runs two operating models. Policies written for the shared clusters can drift from the ones on the dedicated clusters, every tool has to target both, and each “snowflake” cluster handed out becomes the precedent for the next request.

Virtual clusters: cluster-level control on shared nodes

A virtual cluster gives a team its own Kubernetes API server, while its workloads get to keep running on the shared cluster’s nodes.

If we were to suggest any tool for this pattern, it would be vCluster, an open-source virtual cluster implementation.

In vCluster, each virtual cluster runs as a pod or two inside a namespace on the host cluster, holding an API server, a datastore, and a syncer. In return, the tenant gets given admin rights on its own API server, so it can create namespaces, install CRDs and operators, and add its own ClusterRole objects and webhooks, all of which exist only inside the virtual cluster.

When something in the virtual cluster creates a Pod, the syncer copies it into the host namespace, where the host’s scheduler, nodes, CNI, and policies apply to it as they would to any other pod.

In our example, the team that needed the operator would have installed it inside its own virtual cluster. The operator’s CRDs and RBAC would have stayed there, and the platform team would have kept one cluster, one add-on stack, and one upgrade cycle for the nodes.

Virtual clusters do come with their own limits:

  • On shared nodes, isolation is the same as the host namespace. Pods still share nodes and the kernel with other tenants, so the host’s Pod Security Admission, network policies, and quotas are the security boundary, not anything set inside the virtual cluster. Where that is not enough, use a sandboxed runtime, or move the tenant onto vCluster’s private nodes, which give it its own nodes, CNI, and CSI drivers but sit outside the open-source release.
  • Anything that needs the node itself stays with the platform team. In the default shared-node setup, CNI and CSI drivers, GPU device plugins, and node-level security agents run on the host, not inside the virtual cluster.
  • The syncer renames objects when it copies them to the host, so host-level monitoring and logging will show translated names unless they are set up to map them back.
  • Each virtual cluster means one more control plane to version and upgrade, although it runs as a pod or two as opposed to a set of control plane nodes.

Dedicated clusters remain the answer where an audit, a critical service, or an untrusted workload needs a hard boundary. Outside those cases, a virtual cluster gives a team the same control over its own API without adding nodes and another add-on stack to the platform team’s upgrade cycle.

Namespaces by default, virtual clusters for control

Our sane default when designing internal platforms is a namespace per team, a virtual cluster for any team that needs to change cluster-scoped objects, and a dedicated cluster only for critical services, untrusted code, or an audit boundary.

If your estate already looks like the mixed one, much of the work is deciding which dedicated clusters need to exist, and which could be moved back onto a shared cluster as virtual clusters. To start working out which of your clusters could move, book an intro call with LiveWyer. After that, we run a two-hour workshop on your platform with our senior engineers, looking at how your tenants are split today, which of them need cluster-scoped control, and where a virtual cluster would replace a dedicated one. We send you a written summary and recommended next steps afterwards, and we cover the cost of the workshop.

If you are not sure how well your namespaces keep teams apart today, our Kubernetes Security and Compliance review checks how workloads are isolated, alongside access, secrets, and the CVEs being exploited now.

Frequently asked questions

Does Kubernetes have built-in multi-tenancy?

Not as a single feature. The Kubernetes documentation says Kubernetes “does not have first-class concepts of end users or tenants”, so a tenant is whatever you build from namespaces, RBAC, ResourceQuota, NetworkPolicy, and Pod Security Admission.

The same page describes isolation as a spectrum from “soft” to “hard”, and uses “hard” for tenants who do not trust each other. Teams inside one organisation usually sit at the soft end, and the hard end needs separate control planes, dedicated nodes, or a sandboxed runtime on top.

What is the difference between a virtual cluster and a hosted control plane?

Both give each tenant its own API server, and the difference is where the tenant’s pods run. A hosted control plane, as in Kamaji, runs the tenant’s control plane as pods in a management cluster, but the tenant still joins worker nodes of its own, and those nodes bring their own add-ons and upgrades. A virtual cluster syncs the tenant’s pods onto the host cluster’s nodes, so the tenant shares the host’s node pool, CNI, and add-ons.

A hosted control plane makes more sense when tenants need their own nodes anyway; when all they need is their own API, a virtual cluster costs less to run.

Do virtual clusters work on managed Kubernetes such as EKS?

Yes. A virtual cluster runs as pods in a namespace on the host, and vCluster’s documentation lists EKS, GKE, AKS, OpenShift, and self-managed clusters as hosts. On EKS you pay the control plane fee once, for the host cluster, and the virtual clusters on it run on your existing worker nodes.

Thank you for reading

If you enjoy our writing and would like to see more, add us as a Preferred Source on Google and you'll be more likely to see us in your results.

LiveWyer is a London-based consultancy specialising in deploying small, agile engineering teams to design and implement modern Cloud and Platform solutions.

Book an intro call