Running 600 Spot Nodes on GKE

The workload is a headless browser platform: around 1.2 million requests a day, spread across seven browser engines, arriving in bursts that reach 64 times the median hour. Workers are stateless. Each one pulls a job from a queue, drives a browser, writes a result, and pulls the next. Nothing about a single worker is precious. Everything about the fleet's ability to grow from a few dozen nodes to 600 in minutes, and shrink back without paying for the peak all day, is.

That shape is common. Rendering, scraping, media processing, batch inference, test farms. If your work is a queue of independent jobs and your traffic is spiky, this is the playbook, built from first principles so you can rebuild it for your own fleet rather than copy mine.

Part 1: What the workload shape decides for you

Before touching Kubernetes, write down the properties of the work, because they decide every architecture choice that follows.

Jobs are independent and retryable. If a worker dies mid-job, the job can be redone by another worker. This is what makes Spot capacity usable at all. It also means a job must be safe to run twice, which is a property of the job design, not the platform.

Demand is a queue depth, not a CPU number. The right time to add workers is when the backlog grows, before any worker is busy enough for CPU to notice. Scaling on CPU lags; scaling on backlog leads.

The unit of cost is a job, not an hour. Nobody cares what a node costs per hour if they do not know how many jobs that node does per hour. The cost model has to end in cost per thousand jobs, per engine, or it cannot be compared to anything.

Peaks are short and steep. A 64x peak that lasts an hour means the fleet must add hundreds of nodes fast and remove them just as fast. Anything that makes scaling slow, quotas, IP space, slow image pulls, node startup, is a cost problem as much as a capacity problem.

Everything below is a consequence of those four lines.

Part 2: Scheduling from the ground up

Kubernetes places Pods on nodes. That single sentence hides most of what matters at scale.

A Pod declares CPU and memory requests, which is what the scheduler reserves for it, and limits, which is what the kernel enforces. The scheduler bin-packs Pods onto nodes by requests. So the number of worker Pods per node is a function of the worker's CPU request and the node's allocatable CPU. On this fleet a worker requests about half a core, and the measured density is roughly six workers per four-vCPU node once system Pods take their share. That density is a fact you measure from the scheduler, not a number you write in a values file, and it feeds every calculation later.

Nodes come in node pools, groups of identical machines. The cluster autoscaler adds nodes to a pool when Pods are Pending for lack of room and removes them when nodes are empty. That works, but it means someone decides the machine type of each pool in advance, and at 600 nodes across a market where any given machine type can be sold out in a zone, deciding in advance is fragile.

Node auto-provisioning flips that. Instead of predefining pools, you describe what a workload needs and GKE creates and deletes pools to match. The description lives in a ComputeClass, which is where machine routing happens, and Part 6 is about that.

Part 3: Spot capacity, from what it actually is

A Spot VM is unused capacity sold at a discount, up to 91% off on-demand on Google Cloud, with no availability guarantee and no SLA. Compute Engine can reclaim it whenever it needs the capacity back. The discount is the reason the fleet is affordable at 600 nodes; the reclaim is the constraint that shapes how workers are built.

Two facts about reclaim decide the design.

First, the notice is short. When a Spot node is reclaimed, GKE's graceful node shutdown gives your Pods 15 seconds by default to stop, after which critical system Pods get 15 seconds. You can raise the node's shutdown grace period, up to a maximum of 120 seconds total, but a Pod's own terminationGracePeriodSeconds does not extend beyond what the node allows.

Second, reclaim is involuntary and ignores PodDisruptionBudgets. A budget that says "never take more than one of these at a time" protects you from voluntary evictions like upgrades. It does nothing when Compute Engine takes the node.

So the worker has to be designed so that losing it at any instant, with 15 seconds' notice, loses no work. That is a queue-semantics problem:

• The worker acknowledges a message only after the result is written. If it dies first, the message becomes visible again and another worker takes it.
• Jobs are idempotent, so the retry is harmless.
• Long jobs checkpoint or are split, so a 15-second deadline does not throw away an hour.

The last piece is capacity diversity. A Spot pool of one machine type in one zone gets reclaimed as a block when that type is in demand. Spreading across machine families and zones makes reclaims partial instead of total. That is the second reason for the routing ladder in Part 6.

Part 4: SIGTERM, properly

Every scale-down, every deploy, and every Spot reclaim ends the same way: Kubernetes terminates a Pod. If the worker handles that badly, you lose work on every one of those events, and on a Spot fleet they happen hundreds of times a day. Getting this right is the single most valuable piece of worker code.

The termination sequence, from the Kubernetes Pod lifecycle documentation:

1. The Pod is marked for deletion and its endpoints are removed from Services, so no new traffic is routed to it.
2. If a preStop hook is defined, it runs.
3. SIGTERM is sent to process ID 1 in each container.
4. Kubernetes waits up to terminationGracePeriodSeconds, 30 seconds by default.
5. Anything still running gets SIGKILL.

Three consequences for a queue worker.

PID 1 must be your process, or must forward the signal. If the container starts with a shell wrapper, the shell is PID 1 and it will not forward SIGTERM to your worker unless told to. The worker never sees the signal, does nothing for the grace period, and is killed mid-job. Use exec in the entrypoint, or a tiny init, so the signal reaches the code that can act on it.

Draining is a state, not an exit. On SIGTERM the worker should stop pulling new messages, finish or release the job in hand, flush its result, and then exit. "Release" means explicitly making the message visible again rather than waiting for a visibility timeout. On a Spot node the budget for all of that is 15 seconds, so "finish" only applies to work that can complete inside the budget; everything else is released.

The grace period is a promise you have to keep on both sides. Setting terminationGracePeriodSeconds to 300 does nothing on a Spot node that gives you 15. Set the Pod's grace to what the platform actually grants, and build the worker to drain inside it. A worker that needs longer belongs on on-demand capacity or needs shorter jobs.

Concretely, a worker loop that survives all three events looks like this. The shape matters more than the language:

func main() {
	// One context that is cancelled the moment SIGTERM arrives.
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
	defer stop()

	for {
		select {
		case <-ctx.Done():
			// Drain: stop pulling, do not start anything new, exit cleanly.
			return
		default:
		}

		msg, ok := queue.Pull(ctx)
		if !ok {
			continue
		}

		// The job gets the cancellable context too, so a long step
		// aborts instead of running into the SIGKILL.
		if err := run(ctx, msg); err != nil {
			// Release immediately so another worker picks it up now
			// rather than after the visibility timeout expires.
			queue.Nack(msg)
			continue
		}

		// Acknowledge only after the result is durable.
		queue.Ack(msg)
	}
}

Three details carry the reliability. The context is cancelled by the signal, so both the pull and the in-flight job learn about the shutdown without any extra plumbing. The Nack on the way out is what turns a 15-second eviction into a two-second retry somewhere else, instead of a job that sits invisible until its visibility timeout expires. And the Ack after the write, never before, is what makes an abrupt death safe.

The failure mode to watch for is the opposite of a crash. A worker that catches SIGTERM and then keeps working until SIGKILL looks healthy in every dashboard and quietly loses a job on every eviction. Prove the drain works: send a real SIGTERM to a Pod with kubectl delete pod, watch the logs for the drain path, and confirm the message reappears on the queue rather than expiring.

Part 5: Scaling on backlog with KEDA

The Horizontal Pod Autoscaler built into Kubernetes scales on metrics it can see, CPU and memory by default. For queue workers that is the wrong signal: the backlog grows for minutes before any worker's CPU moves, and the fleet is late every time.

KEDA is the standard fix. It reads an external metric, here the number of undelivered messages on a Pub/Sub subscription, and turns it into scaling decisions. The architecture is worth understanding because its failure modes follow from it.

KEDA owns the zero-to-one decision. The HPA owns the rest. A ScaledObject names a Deployment and a trigger. KEDA's operator watches the metric and decides whether the workload should be active at all: if the metric is above the activation value, it scales from zero to the minimum; if it falls to zero, it scales back to zero. In between, KEDA generates a standard HPA and feeds it the metric through a metrics adapter, and the HPA does the arithmetic of replicas equals backlog divided by target-per-Pod.

That split matters in two ways. Scaling to zero is KEDA's feature, and it is what makes an engine that gets no traffic for an hour cost nothing. And every scaling decision above one replica is an HPA decision, with the HPA's own stabilisation and rate behaviour, so tuning happens in two places.

The trigger for Pub/Sub defaults to the subscription's undelivered message count, with a target of 10 messages per Pod and an activation value of 0, and reads it from Cloud Monitoring over a short time horizon, which means the number lags the queue by a minute or two. Choose the target per Pod from measured throughput: if a worker clears about one job every 20 seconds, ten queued messages per worker is roughly three minutes of backlog per Pod, which is a reasonable lead time for a two-minute node startup.

Worked example: what a burst actually asks for. The HPA's arithmetic is desired replicas = ceil(current metric / target per Pod). Follow a backlog of 40,000 messages through it:

Step	Arithmetic	Result
Desired replicas	40,000 / 10 per Pod	4,000
Capped by the ScaledObject maximum	min(4,000, 400)	400
Nodes needed at 6 Pods per node	400 / 6	67
Time to serve the backlog at 3 jobs/min/Pod	40,000 / (400 x 3)	~33 minutes

Three lessons fall straight out of that table. The maximum replica count, not the queue, is what decides your worst-case drain time, so it is a latency setting wearing a capacity costume. Multiplying it by Pods per node gives the node count you must have address space and quota for, which is the arithmetic in Part 6. And if 33 minutes is too slow, the lever is either a higher cap or faster workers, not a smaller target per Pod, because lowering the target only makes the HPA ask for replicas it is not allowed to have.

The reverse direction has its own trap. Once the backlog drains, the metric falls and the HPA scales down, but the cooldown before a workload is allowed back to zero is measured in minutes, and every node removed is a node you pay to start again if the next burst is close. On a spiky workload it is usually cheaper to hold a small floor of warm capacity than to pay the startup latency on every peak. Measure the gap between bursts before choosing.

The metric source is a single point of failure. On this fleet the queue depth for every ScaledObject flows through one small proxy service that reads Pub/Sub and Cloud Monitoring and exposes the numbers. If that proxy is down, KEDA cannot see backlog, no engine scales on demand, and nothing in the default setup alerts about it. Treat the metrics path with the same seriousness as the workers: give it generous CPU, a probe timeout that survives a busy node, and an alert on its absence. Its load also scales with the number of subscriptions it polls, which is invisible from the values file that adds a subscription. Measure it after every fan-out change.

Ceilings should have one source of truth. A fleet has several caps that must agree: the HPA maximum, any admission gate that refuses work when the backlog is too deep, and the queue's own age limit after which jobs expire. When those numbers live in three places they drift, and the failure is subtle: a gate sized for an old fleet quietly refuses work the new fleet could have done. Derive them from one value in the chart, so a typo fails the render instead of throttling production.

Part 6: Routing across machines, and the two ceilings

At a few dozen nodes, you pick a machine type. At 600, the market picks for you. Any given type in any given zone can be sold out at the moment you need 200 more, and Spot makes that more likely, not less. So the fleet is described as a preference order, not a type.

The ComputeClass ladder

A ComputeClass is a list of priority rules. Each rule names a machine family, whether it is Spot, and optionally more. When a Pod needs a node, GKE tries the rules in order and creates a node from the first one the cloud can satisfy. If none can, whenUnsatisfiable decides whether to fall back to the cluster default or leave the Pod Pending. With node pool auto-creation enabled, GKE creates and deletes the pools itself. With active migration enabled, it later moves Pods back to a higher-priority rule when that capacity returns, so a stockout does not leave you on your fourth choice forever.

Building the ladder is a pricing exercise. I priced thirteen machine families against the Cloud Billing Catalog API for the regions the fleet runs in, at Spot rates, per unit of the resource the workers actually consume, and ordered the rules from cheapest-per-job upward. A more expensive family lower on the ladder is not a mistake; it is the price of getting capacity in the ten minutes that matter.

Ceiling one: capacity

When a rule cannot be satisfied, the cloud says so with a stockout error, and the ladder falls through to the next rule. This is the failure the ladder is designed for, and it handles it well. During one peak, the first family was refused a dozen times and three lower rules absorbed the demand.

Ceiling two: addresses, which the ladder cannot fix

Every node gets a block of Pod IP addresses carved from the cluster's Pod range, and the block size is set by the maximum Pods per node. GKE assigns a block with at least twice as many addresses as the max Pods, rounded up to a power of two. The default of 110 Pods per node yields a /24, 256 addresses, per node. A cluster with a /16 Pod range, 65,536 addresses, therefore tops out at 256 nodes. Set max Pods to 32 and each node takes a /26, 64 addresses, and the same range holds 1,024 nodes.

That setting is fixed when a node pool is created, and pools created by auto-provisioning use the cluster default unless the ComputeClass says otherwise. So the arithmetic is: measure real Pod density, on this fleet about six per node, set max Pods with a comfortable multiple, here 32, and put it in the ComputeClass template so no rule can omit it. Address space is cluster-wide, so no amount of machine-family fallback helps when it runs out. Every rung dies on the same error. Do the division before you need it.

Max Pods per node	Block per node	Nodes on a /16	Nodes on a /15
110	/24	256	512
64	/25	512	1,024
32	/26	1,024	2,048

Quotas, and which one binds first

Cloud quotas are per resource, and the one that stops a Spot fleet is rarely the one you expect. On this fleet each worker node carried a 100 GB boot disk, and the regional quota on total persistent disk gigabytes was 30,000. That is 300 nodes, regardless of CPU quota. The fix was free: halve the boot disk, which the workers used a tiny fraction of, and the ceiling doubled. Then raise the disk and preemptible CPU quotas in advance of the next growth step, not during it.

The general method: list every quota a node consumes, disk, CPU by family, preemptible CPU, instances, addresses, divide each by the per-node consumption, and the smallest result is your real ceiling. Write it down next to the IP arithmetic. Both are capacity numbers that no dashboard shows you until you hit them.

Part 7: Engineering the cost

Cutting compute cost 30% on this fleet came from a sequence of measurements, not a single trick.

Start with a unit. Cost per thousand jobs, per engine. That requires two feeds: the billing export, sliced by the labels that identify each engine's node pools, and the job counts from the queue. Once the unit exists, every optimisation can be checked against it. Without it, "we saved money" is a feeling.

Why the unit is not optional, with illustrative numbers in the shape the real ones took:

	Family A	Family B
Spot price per node-hour	$0.042	$0.061
Jobs completed per node-hour	310	560
Cost per 1,000 jobs	$0.135	$0.109

Family A is 31% cheaper per hour and 24% more expensive per job. Every hourly-price comparison would have picked the wrong machine, and at fleet scale that error compounds across every node for as long as nobody recomputes it. The reason is usually mundane: newer cores finish a browser job faster, so the more expensive node is busy for less of the hour per unit of work.

The same table is how you decide whether an optimisation was real. A change that cuts the hourly bill but slows workers can leave cost per thousand jobs flat or worse, and only the unit shows that.

Price the machines properly. The Catalog API gives list prices per SKU per region, on-demand and Spot. Normalise to the resource the workers are bound by, then rank. This produced the ladder, and it exposed families that looked cheap per hour and were expensive per job because their Spot prices in the fleet's regions were poor.

Right-size the things nobody looks at. Boot disks sized at 100 GB for a workload that uses under 2 GB were both a quota cap and a line item. Storage lifecycle rules on the buckets that hold results, because results a month old are never read. NAT egress, because a fleet that pulls images and talks to APIs through a NAT gateway pays per gigabyte, and the fix is private access to Google APIs plus a registry mirror in-region.

Model burst separately from baseline. A fleet with a 64x peak has a real bill that is mostly burst and egress, not the fixed baseline that a naive model estimates. Any "move it somewhere cheaper" comparison has to price the burst honestly, including whether the cheaper place can supply 400 nodes on demand, or the saving is fictional.

Don't price engineer time into infrastructure comparisons. It sounds prudent and it is usually a thumb on the scale. Compare infrastructure to infrastructure, and treat the operational model as a separate decision with its own arguments.

Lever	What it changed	Trade
Spot ladder ordered by cost per job	Largest single saving	Reclaims; handled in Parts 3 and 4
Machine family pricing via Catalog API	Removed poorly priced families	Maintenance as prices move
Boot disk 100 GB to 50 GB	Cost and a doubled quota ceiling	Node pool replacement to apply
Storage lifecycle rules	Steady-state storage bill	Old results are gone
Private API access and registry mirror	NAT egress	Setup once

Part 8: Delivering changes to 600 nodes without touching them

At this size, the deployment model is a reliability feature.

Terraform owns the cloud. Cluster, node pool defaults, networking, IAM, quotas as code. Nobody clicks.

Helm owns Kubernetes. One chart, one production values file. Two values files that describe different clusters and overwrite each other on alternate deploys is an incident waiting to happen, and it happened once. One file, reviewed, is the rule.

Argo CD owns the difference. It renders the chart from the repository and continuously reconciles the cluster to it, with auto-sync, self-heal and prune. Self-heal means a hand-edited resource is reverted; prune means a resource removed from git is removed from the cluster. Together they make kubectl apply not just discouraged but pointless, which is the goal. Image updates flow the same way: a merge builds an immutable tag, an image updater notices it, and the rollout happens through git, not a terminal.

Progressive delivery for the workers. Argo Rollouts replaces a Deployment with a staged rollout: a small fraction of Pods on the new image, a pause, an analysis of error rate and job throughput, then the rest. On a fleet where a bad image can burn through a hundred thousand jobs in an hour, the analysis step is not ceremony.

Verify after every change. A script that checks the live values against the rendered chart, the rollout status, and a real request through the platform. Argo CD saying Synced and Healthy is necessary, not sufficient.

Part 9: Seeing the fleet without drowning in it

Observability at 600 nodes fails in a specific way: the pipeline that is supposed to show you the problem becomes the problem.

Metrics identify the engine, not the Pod. A Prometheus time series is a metric plus a unique set of labels. On an autoscaling fleet, Pod names never repeat, so a pod label multiplies series by churn, not by size, and dead Pods' series stay in memory for two to three hours after they are gone. Nobody on call has ever asked "what failed on pod worker-abc12", because that Pod no longer exists by the time they look. They ask "what failed on engine X". So the pod label is dropped at scrape time, aggregation is by engine and subscription, and per-Pod detail lives in logs, where a Pod name is content rather than a dimension.

Dashboards aggregate. Per-Pod scraping means every restart starts a new series; a panel that plots raw series shows ghosts. max by (subscription) and rate() on counters. The series count on one dashboard dropped from 31 to the correct 7 with that one change.

Alerting must route somewhere a human looks. Rebuilding alerting was less about thresholds and more about routing: engine-level symptoms to a channel the on-call reads, infrastructure noise to a channel nobody has to, and an explicit alert on the KEDA metrics path being absent, because that failure is silent by default.

Logs are for the individual. Loki with a few labels, and the Pod name inside the line. Cheap at that granularity, and exactly where a per-Pod question belongs.

Part 10: Keeping it reliable

The practices that hold the rest together are unglamorous.

Capacity numbers must track reality. A capacity file that derives "1,024 nodes of headroom" from a setting on node pools that are no longer the ones in use is worse than no file. After any change to how nodes are created, find every derived number and re-derive it. The IP arithmetic in Part 6 was learned this way.

Postmortems on near-misses, not just outages. A stalled autoscaler that recovered in an hour still gets a written cause, a fix, and a check that the fix cannot silently regress. That is where "put the default in the template so no rule can omit it" came from.

Base images are a security surface at fleet scale. A vulnerability finding in a base image is multiplied by every image built on it. Moving the fleet's images to a base whose maintainers actually patch the packages you ship took most findings to zero, and the audit that required it cared about the registry scanner's numbers, not a local tool's.

Docs are claims to verify. Every comment that says "3 pods per node" or "bound by this quota" is a hypothesis. Measure live, then fix the comment. The stale ones were wrong in ways that would have bought hardware for the wrong reason.

What this actually teaches

The fleet works because every layer is designed around the same four facts from Part 1. Jobs are retryable, so Spot is safe and SIGTERM is a drain rather than a disaster. Demand is a backlog, so KEDA leads instead of lagging. Cost is per job, so the machine ladder and every optimisation are measured against one unit. Peaks are steep, so the ceilings that stop growth, capacity, addresses and quotas, are computed in advance rather than discovered at 253 nodes.

None of the individual pieces is exotic. Kubernetes scheduling, Spot semantics, signal handling, an external autoscaler, a priority list of machines, some division, a billing export, GitOps, and a metrics pipeline that knows which label matters. The skill is in how they fit, and in insisting on a measured number at every seam where a guess would have been easier.

Continue learning

If the scaling model now makes sense, the next thing to understand is the HPA's own behaviour in depth, stabilisation windows and scaling policies, because KEDA hands it every decision above one replica. After that, Pod termination and graceful shutdown at the kubelet level, which is where the 15-second budget really lives. Then Prometheus TSDB internals, so the cardinality rule in Part 9 becomes obvious rather than remembered. Those three cover most of what a large autoscaling fleet will ever do to you.

Sources and further reading

Primary sources for the claims above, and the material worth a full read.

• Spot VMs in GKE and Compute Engine Spot VMs. The reclaim mechanics, the 15-second budget, the 120-second ceiling, and the fact that PodDisruptionBudgets do not apply.
• Kubernetes: Pod lifecycle and termination and Node graceful shutdown. The exact sequence behind Part 4, and how the kubelet splits shutdown grace between regular and critical Pods.
• Kubernetes: Horizontal Pod Autoscaling. KEDA hands every decision above one replica to this controller, so its stabilisation windows and scaling policies are your tuning surface.
• KEDA: Scaling deployments and the GCP Pub/Sub scaler. The activation versus scaling split, the ScaledObject fields, and the metric lag caveat.
• GKE: About custom compute classes. Priority rules, whenUnsatisfiable, active migration and node pool auto-creation.
• GKE: Configure maximum Pods per node and Google's post on avoiding the IP_SPACE_EXHAUSTED error. The per-node block rule and the address arithmetic in Part 6.
• EKS: Choosing an instance type and Assign more IP addresses with prefixes. The same ceiling on AWS, and how prefix delegation changes it.
• Kubernetes scalability thresholds from SIG Scalability. The tested limits of the control plane, for knowing which ceilings are Kubernetes' rather than the cloud's.
• Cloud Billing Catalog API. Where the machine-family pricing in Part 7 comes from.
• Argo CD documentation and Argo Rollouts. Auto-sync, self-heal, prune, and progressive delivery with analysis.
• Helm: Custom resource definitions. Why crds/ and templates/ behave differently on uninstall.
• Google SRE book: Postmortem culture. The practice behind Part 10, from the people who scaled it.

FAQ

Is Spot safe for anything stateful? Not with this design. Everything here assumes a job can be lost at 15 seconds' notice and redone. Stateful services belong on on-demand capacity, and the ComputeClass ladder can express that per workload.

Why not scale on CPU with a low threshold? Because backlog grows before CPU does, and a queue that is already deep when scaling starts has already blown its latency budget. Scale on the thing users wait for.

How do I know my real node ceiling? Take the Pod range size divided by the per-node block from max Pods per node, and take each quota divided by per-node consumption. The smallest of those is the ceiling. Compute it before the fleet is a fifth of the way there.

Does the machine ladder replace capacity reservations? It reduces the need for them. If a workload truly cannot tolerate a stockout, a reservation for a baseline plus a ladder for burst is the combination, and the ladder's whenUnsatisfiable setting decides what happens when even that runs out.