CapacityQuota: limiting what the autoscaler provisions, not what pods request

A new cluster-scoped CRD lets you cap autoscaling by node attribute — machine family, accelerator type, zone, ComputeClass — rather than by namespace. It closes a gap ResourceQuota was never shaped to fill, and it lands upstream in kubernetes/autoscaler, not only in GKE.

Every platform team eventually writes some version of this ticket: “finance approved a 100-core commitment on N4. Make sure we don’t autoscale past it — but don’t let the workloads go Pending either.”

It sounds like a quota problem, so you reach for ResourceQuota, and then you discover it can’t express the sentence. Not because you configured it wrong, but because ResourceQuota is answering a different question than the one you asked. You end up encoding the limit into node pool topology instead: a separate pool with a max size, a separate class, some weight, and a comment explaining that this shape exists because of a contract signed in a different building.

CapacityQuota is the mechanism that lets you say it directly. It’s available on GKE 1.36.2-gke.2064000+ (Standard and Autopilot), and — the part I find more interesting — it isn’t a GKE-shaped extension. It’s an upstream Cluster Autoscaler API.

Two different questions

#

The reason ResourceQuota can’t do this isn’t a missing field. It’s a scope mismatch, and it’s worth being precise about, because the two objects sound interchangeable and aren’t.

ResourceQuota asks: may the pods in this namespace request this much? It’s namespace-scoped, it counts resources.requests and resources.limits on pod specs, and it enforces at admission — exceed it and pod creation is rejected, with an error at kubectl apply time.

CapacityQuota asks: may the autoscaler provision this much of this kind of node? It’s cluster-scoped, it selects nodes by label, and it enforces during scale-up evaluation — exceed it and no node is created, with pods sitting Pending until capacity appears somewhere else.

The upstream proposal puts the structural version of this in one line: ResourceQuota and LimitRange are namespace-scoped, “while nodes are global”. A node isn’t in a namespace. Pods from six namespaces land on it. So no namespace-scoped object can express “at most 100 cores of N4 exist in this cluster,” because the noun it governs isn’t the noun you’re trying to limit.

There’s a subtler problem too, and it bites even if you’re willing to approximate. Pod requests are not node capacity. The autoscaler bin-packs pods onto node shapes, and the proposal is blunt that it “does not guarantee that bin packing will yield the optimal result” — plus every node loses some capacity to system daemons and OS overhead. So a 100-core ResourceQuota across your namespaces does not produce 100 cores of nodes. It produces somewhere north of that, by an amount that depends on pod shapes you don’t control. As a way to hit a commitment number, it’s an estimate wearing the costume of a limit.

The existing autoscaler knobs don’t close it either. Cluster-wide cores/memory limits and --max-nodes-total are real, but they’re cluster-wide — one number for everything, no way to say “this applies only to N4.” Per-node-pool min/max works, but only if you’ve already fragmented your topology so that the thing you want to limit happens to live in its own pool. Which is exactly the workaround we’re trying to stop writing.

The API

#

Two fields. A selector over node labels, and a set of limits.

apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityQuota
metadata:
  name: n4-quota
spec:
  selector:
    matchLabels:
      cloud.google.com/compute-class: prefer-n4
      cloud.google.com/machine-family: n4
  limits:
    resources:
      cpu: 8

The selector is a standard Kubernetes LabelSelector, so matchExpressions works too, with In / NotIn / Exists / DoesNotExist — the same vocabulary as nodeAffinity. Omit the selector entirely and the quota applies to every node in the cluster.

limits.resources accepts cpu, memory, nodes, nvidia.com/gpu, and google.com/tpu. Two things to know before you write one:

• Integers only. Milli units are not supported — cpu: 8 is fine, cpu: 8500m is not. This is a limit on node capacity, and fractional cores of a node aren’t a meaningful unit at this layer.
• nodes is a first-class key. For heterogeneous priority ladders where CPU counts vary per rung, capping node count is sometimes the honest expression of what you mean.

And one operational detail that will save you a confused afternoon: a CapacityQuota is only enforced while its cluster-autoscaler.kubernetes.io/valid condition is True. An invalid quota isn’t a loud failure — it’s a silently inert object. Check it before you believe it:

kubectl describe capacityquota n4-quota

Where it pairs with ComputeClasses

#

I’ve written three posts recently about ComputeClasses as an ordering API: you author a ranked list of what you want, the autoscaler walks it top-down, and each rung either succeeds or fails cleanly so the next one gets its turn. Part 3 closed with a brief version of the pattern below, as the third layer of tenant isolation. Here’s the thing I under-sold there.

A ComputeClass priority ladder expresses preference. It has no way to express quantity. You can say “N4 first, then N4D, then C4,” but there is no field on a priority rung that means “…and stop at 100 cores of N4.” The ladder moves on when a rung can’t be satisfied — a stockout, a quota error from the cloud — not when you’ve had enough of it.

That’s the gap CapacityQuota fills, and it’s why the two objects are better together than either is alone. The quota supplies the missing dimension: how much of each rung.

The mechanism is entirely in the selector. Because the quota matches on cloud.google.com/machine-family: n4, N4D and C4 nodes simply don’t carry that label — so they’re outside the quota, uncapped, and available. Scale-up on N4 stops at the ceiling with a noScaleUp:

Pod didn't trigger scale-up: 1 exceeded quota: "CapacityQuota/n4-quota", resources: cpu

…and the pending pods spill down the ladder onto N4D and C4 instead of waiting. No Pending backlog, no second ComputeClass, no topology surgery. There’s a runnable version of exactly this in my examples repo — three manifests and a deployment of 16 replicas sized to overshoot the cap on purpose, so you can watch the spillover happen rather than take my word for it.

What you’ve written, in two objects, is: “use exactly as much of my discounted capacity as I bought, then keep going on undiscounted capacity.” That’s a sentence a finance partner recognizes, expressed declaratively, in the cluster, where it can actually be enforced.

The label surface is the policy surface

#

The machine-family cap is the demo, but it’s one instance of something more general. Any node label is a limit dimension, and you can combine them.

That yields quotas that were previously awkward to express at all:

# Cap a specific accelerator across the whole fleet
selector:
  matchLabels:
    cloud.google.com/gke-accelerator: nvidia-tesla-t4
limits:
  resources:
    nvidia.com/gpu: 16

# Blast-radius limit: no more than 64 nodes in any one zone
selector:
  matchLabels:
    topology.kubernetes.io/zone: us-central1-b
limits:
  resources:
    nodes: 64

# Cap the expensive families as a group, leave everything else alone
selector:
  matchExpressions:
  - key: cloud.google.com/machine-family
    operator: In
    values: [c2, c3, c3d]
limits:
  resources:
    cpu: 64
    memory: 128Gi

The proposal’s framing of this is the right one: “adding a new dimension for limits only requires ensuring the nodes are labeled appropriately.” You’re not waiting on an API change to cap a new axis. If you can label it, you can limit it.

One deliberate exception: GKE rejects quotas that select on node.kubernetes.io/instance-type or beta.kubernetes.io/instance-type. Cap the shape through a ComputeClass instead. That’s a slightly opinionated restriction, and I think it’s the right one — instance-type-level quotas are how you end up back in topology management, which is the problem this was built to get you out of.

What it deliberately doesn’t do

#

This is a limit on scale-up evaluation. Read that narrowly, because several consequences follow and each one has surprised somebody:

• No retroactive enforcement. Lower a limit below current usage and nothing gets reclaimed. The autoscaler will not scale down nodes that already exceed the cap; it only declines to create more. A tightened quota takes effect at the pace of your natural churn.
• Soft, not hard. The docs say limits “aren’t strictly enforced” and small margin overages can occur. If you need a hard ceiling for a compliance reason, this is the wrong instrument — cloud-level quota is.
• Manual scaling bypasses it. Resize a node pool by hand and the quota doesn’t stop you. It governs the autoscaler, not the cluster.
• Active migration can fail against it. If activeMigration wants to move workloads back to a preferred rung but the resulting nodes would violate a quota, the migration doesn’t happen. That interaction is worth thinking through before you cap the rung you’ve told the cluster to prefer — you can otherwise write a pair of objects that quietly argue with each other forever.
• status.used is observational. It reflects capacity after successful scale-ups. The autoscaler tracks pending nodes internally during evaluation, so don’t build tooling that treats status.used as the enforcement ledger.

Minimums are also explicitly out of scope — the proposal removed them and points at the separate CapacityBuffer API for “keep this much warm” semantics. Worth knowing so you don’t go looking for a min field that was considered and cut.

The part I actually want to flag

#

I closed the Karpenter series by saying GKE’s autoscaling stack is opening up and that a fuller roadmap was coming. This is a concrete instance of what that looks like in practice, and it’s more interesting than a GKE feature announcement.

CapacityQuota isn’t a Google API that might get contributed upstream someday. It is the upstream API. The design lives in kubernetes/autoscaler as proposals/granular-resource-limits.md, the group is autoscaling.x-k8s.io — the SIG-Autoscaling group, not cloud.google.com — and the implementation sits in Cluster Autoscaler behind --capacity-quotas-enabled. It went alpha in Cluster Autoscaler v1.35; GKE ships v1beta1 today, ahead of upstream’s v1alpha1.

Two details in that proposal are worth reading as signals rather than trivia.

First, the motivation section benchmarks against Karpenter as a peer, noting it supports limits on a NodePool but has no cluster-wide equivalent. That’s the same comparison I spent three posts making, being made in the upstream design doc — which is a healthier place for it to happen than a vendor blog.

Second, the graduation criteria. GA is targeted at v1.39, and it’s noted as pending Karpenter adoption. Read that carefully: the bar for this API is not “GKE ships it,” it’s “the other major provisioner adopts it too.” An API that both Cluster Autoscaler and Karpenter implement is a portability primitive — a limit you write once and carry between clouds, rather than another dialect to translate.

That’s the version of open-sourcing that actually matters to someone running two platforms. Not “the code is readable,” but “the object I wrote works on the other side.” It’s early — alpha upstream, Preview on GKE, and the Karpenter half is an aspiration in a markdown file rather than a shipped feature. But it’s the first piece of this stack where the portability story is being written into the design instead of reconstructed afterwards by people like me.

If you’re running a commitment you’re trying not to overshoot, it’s worth twenty minutes with the examples repo this week.