Challenge
Your cloud bill has grown faster than the workloads running on it, and the reasons tend to be the same few. If one of these sounds like yours, or a Savings Plan or reservation is due for renewal, a review of your cloud spend is the place to start.
Clusters sized for peak
Node groups were sized for peak traffic and stay that size overnight and at weekends, and pod requests ask for far more CPU and memory than they use, so clusters look full while most of their CPU sits idle. Spot was ruled out after one interruption, so everything runs on demand.
Spend outside the clusters
Instances and managed databases are still on the sizes they were given at launch, volumes and snapshots have outlived their servers, and traffic between availability zones and through NAT gateways adds charges nobody planned for.
Nobody owns the spend
Several teams share each cluster, so the bill shows nodes as opposed to the services running on them, and a lot of cloud resources carry no team tag. When finance asks which team is responsible, nobody can say who.
Commitments which no longer fit
Savings Plans or reservations were sized for the estate you used to run, so part of the commitment now goes unused, or nothing was ever committed and the steady baseline still runs at on-demand rates.
Approach
Your cloud provider’s recommendations, or a cost tool, will list the instances and pods which look oversized. We do the rest: deciding which workloads can run on Spot, changing requests in your own repositories, and getting each change into production. We pair with your engineers as we go, so they can carry on the FinOps work after we leave, and we run hands-on training if they want to go further.
The work starts with an intro call and a two-hour workshop on your platform. If cost is the main problem, we review your cloud spend, in the clusters and across the rest of the account, and give you a written plan with each change ranked by the saving we expect and the effort it takes, plus the carbon it saves if you report on emissions.
| Rank | Change | Saving | Effort |
|---|---|---|---|
| 1 | Move node groups to Karpenter | 5 out of 5 | Medium |
| 2 | Right-size pod requests | 4 out of 5 | Low |
| 3 | Spot for stateless services | 3 out of 5 | Low |
| 4 | Savings Plan for the baseline after the rest | 3 out of 5 | Low |
- $2.4m
- saved each year at Siemens
- 130
- clusters moved to Karpenter
- 6,000+
- engineering hours saved each year
What does Kubernetes and cloud cost optimisation involve?
Cost optimisation is the work of matching the capacity you pay for to the capacity your workloads use, and of moving workloads onto cheaper ways of running, such as Spot capacity, commitments, or services which scale to zero when idle. The saving usually comes from some mix of the following:
Workload right-sizing
CPU, memory, and GPU requests set from measured usage, and instances and managed databases sized the same way, which also cuts software licences charged per core
Nodes and clusters
Karpenter or the Cluster Autoscaler fitting instance types to pending pods, consolidating mostly empty nodes, scaling non-production down out of hours, and merging underused clusters
Spot and commitments
Interruption-tolerant workloads on Spot instances, spread across instance types and availability zones, and Savings Plans or reserved instances for the steady baseline left after right-sizing
Storage and networking
Deleting unattached volumes and old snapshots, moving data which is rarely read to cheaper storage classes, and cutting traffic between availability zones and through NAT gateways
Logs and metrics
Log retention set to how far back anyone searches, debug logging switched off in production, and metrics trimmed of the high-cardinality labels which multiply ingestion charges
Cost per team
Spend reported per team, namespace, or application from consistent labels and tags, as showback or chargeback, so every line of the bill has an owner
Not always worth paying for
If your cloud bill is small, or your clusters and instances already run close to the capacity you pay for, the review may cost more than it saves, and we will tell you that in the intro call.
Kubernetes and Cloud Cost Optimisation Roadmap
Cost work moves through four stages, and you decide after the review which changes are worth making.
Talk
What happens:
- Tell us what you run
- Review your platform over a screen share
- Send a written summary
What you leave with:
- Whether a cost review is worth it
- Recommended next step
Review
What happens:
- Compare requests and sizes with usage
- Check node provisioning and Spot fit
- Trace spend to teams
What you leave with:
- Written plan of changes
- Expected saving and effort for each
- How much of the bill each team owns
Change
What happens:
- Make the changes you choose
- Roll out one cluster, node pool, or service at a time
- Measure the saving from each
What you leave with:
- Changes in production
- Cost per team reporting
Measure
What happens:
- Compare the bill with where you started
- Check each saving against its estimate
- Size commitments from the new baseline
What you leave with:
- Saving per change against its estimate
- Commitments sized to what you now run
- What is left to tackle
Case studies
Kubernetes and Cloud Cost Optimisation FAQ
Leila Haddad, Chief Financial Officer
How much could we save?
LiveWyer
We estimate it in the review, change by change
We do not quote a percentage before we have seen your cloud spend, because the saving depends on how over-provisioned your clusters and instances are, how many workloads can tolerate Spot capacity, and how much of your steady usage already has a commitment. The review gives you an estimate for each change, so you can decide which ones are worth the effort. The review itself is scoped in the workshop, and you get an estimate of its cost before you commit.
Owen Pritchard, Head of Infrastructure
Is Spot capacity safe for production workloads?
LiveWyer
For workloads which tolerate interruption, yes
Stateless services with several replicas usually run well on Spot, as long as they have pod disruption budgets and the node pool can choose from a wide range of instance types. Karpenter watches for Spot interruption notices and drains a node before it is reclaimed. Databases, stateful services which are slow to recover, and long batch jobs which cannot checkpoint are better kept on on-demand or reserved capacity, and the review says which of your workloads fall into which group.
Katie Brennan, Head of Platform Engineering
We already use the Cluster Autoscaler. Do we need Karpenter?
LiveWyer
Not always
If your clusters run a few node groups with workloads of similar sizes, right-sizing requests and tuning the Cluster Autoscaler may get most of the saving with less change. Karpenter pays off when workloads vary in size, when you want Spot capacity across many instance types, or when node groups and AMI updates have become a burden to manage, as they had at Siemens. The review recommends one or the other for each cluster.
Anjali Rao, Enterprise Architect
Do you only look at Kubernetes?
LiveWyer
No, but it is where we go deepest
The review also covers instances, managed databases, storage, load balancers, data transfer, logs and metrics, and commitments across the rest of your cloud account, on AWS, Azure, or GCP. Our largest saving was on Amazon EKS, and the same cluster work applies to AKS and GKE: on AKS, node auto-provisioning is built on Karpenter, and GKE has its own node auto-provisioning and Spot VMs. On clusters on premises, the saving comes from right-sizing and consolidation as opposed to Spot capacity.
Samuel Adeyemi, Operations Manager
Will the changes cause downtime?
LiveWyer
They should not
Changes roll out one cluster, node pool, or service at a time through your pipelines, so each one can be rolled back, and pod disruption budgets are in place before nodes start being replaced. Resizing a managed database usually means a restart or a failover, so we schedule those with you. Across 130 Siemens clusters, the move to Karpenter caused no downtime and no customer impact.
Emma Lindqvist, Engineering Director
How do we see which team is spending what?
LiveWyer
Labels, tags, and a cost tool
Spend can only be split by team if workloads carry consistent labels or sit in namespaces each team owns, and cloud resources carry consistent tags. Once they do, a tool such as OpenCost can report cost per team, namespace, or application, your cloud provider's cost reports can do the same for tagged resources, and the numbers can go to the people who can act on them. We set up the labelling, tagging, and reporting as part of the work, and where you already use a commercial tool, we work with that.
Priya Shah, FinOps Lead
How do we stop the costs creeping back?
LiveWyer
Policies in the clusters and alerts for each team
We add policies to the clusters, with a policy engine such as Kyverno, which reject new workloads that have no resource requests or team labels, so the next service cannot undo the right-sizing. Budgets and spend alerts go to the team which owns each part of the bill, so a jump reaches the people who caused it instead of finance at the end of the month. The Measure stage then checks each saving against its estimate, and anything which has crept back goes into the next round of changes.
Book an intro call
Want to know where your cloud spend goes? Book an intro call of up to an hour with one of our senior engineers, and we will tell you whether a cost review is worth it. If it is, we follow up with a two-hour workshop on your platform, at our cost.
Book an intro call