What you'll learn
Quick Answer
Autoscaling watches a metric like CPU, memory, or queue depth on a fixed interval, then adds or removes capacity to keep that metric near a target. Horizontal scaling (scale-out) adds more instances or pods; vertical scaling (scale-up) makes an existing one bigger. Because measuring, deciding, and booting new capacity all take real time, autoscaling always reacts a little after the fact — managing that lag with cooldowns and stabilization windows is the actual engineering problem, not eliminating it.
What Autoscaling Actually Watches
An autoscaler doesn't magically sense that your app is under load. It polls one or more specific metrics on a fixed schedule and compares the reading to a target you set. What metric you pick changes how well the whole system behaves, so it's worth being precise about the options.
- CPU utilization is the default almost everywhere because every VM and pod already reports it for free. It's a poor signal for I/O-bound services, though — a process waiting on a slow database call or an external API isn't burning CPU, so a service that's actually struggling under load can show low CPU the whole time.
- Memory matters for JVM-style runtimes with real GC pressure, but it's risky as your only trigger: memory doesn't always fall back down once a spike passes (caches, connection pools, and leaks all hold onto it), so a memory-triggered scale-out can add capacity that never scales back in.
- Custom metrics — requests per second, p99 latency, queue depth in SQS or RabbitMQ — are usually the most accurate signal for real user-facing load. Kubernetes gets these through adapters like the Prometheus Adapter or event-driven scalers like KEDA; AWS gets them through custom CloudWatch metrics.
None of this is measured continuously. By default, EC2 basic monitoring reports metrics like CPU utilization to CloudWatch every 5 minutes; you can pay extra for detailed monitoring to get 1-minute granularity instead. Kubernetes' HPA controller runs its own separate polling loop on top of whatever metrics-server has cached. That polling interval is the very first source of lag baked into every autoscaler, before any decision even gets made.
Horizontal (Scale-Out) vs Vertical (Scale-Up) Scaling
Scale-out (horizontal) means adding more instances or pods running identical code behind a load balancer. It requires the workload to be effectively stateless per instance — session data has to live somewhere external (Redis, a database), and nothing important can live only on one instance's local disk — because any instance can be killed or added at any moment.
Scale-up (vertical) means giving the existing instance more CPU, RAM, or IOPS: resizing an EC2 instance to a bigger type, or raising a Kubernetes pod's resource requests through the Vertical Pod Autoscaler. There's no load-balancing complexity and no distributed-state problem to solve, but there's a hard ceiling (the largest instance type that exists) and it almost always needs a restart — a VPA resize restarts the pod at its new size, and resizing an EC2 instance means stop, change instance type, start again. That means vertical scaling can't respond to a live traffic spike; it's a capacity-planning tool, not a real-time reflex.
The practical split: horizontal scaling is the default for stateless web and API tiers because it has almost no single-node ceiling, at the cost of real distributed-systems overhead. Vertical scaling still wins for single-writer databases, in-memory caches that resist trivial sharding, and legacy monoliths that were never built to run more than one copy at a time.
Inside the Control Loop
Autoscaling is a control loop, not an event handler: observe the metric, compute the desired capacity, compare it to current capacity, act if they differ, wait, repeat. It never reacts to a single instantaneous reading; it reacts on its own schedule.
Kubernetes' Horizontal Pod Autoscaler uses a specific, published formula:
desiredReplicas = ceil( currentReplicas * ( currentMetricValue / desiredMetricValue ) )Worked example: 4 replicas are currently averaging 80% CPU against a target of 50%.
ceil(4 * (80 / 50)) = ceil(4 * 1.6) = ceil(6.4) = 7 replicasThat formula only gets evaluated on a fixed cadence, not continuously. The HPA controller inside kube-controller-manager runs on a sync period controlled by the --horizontal-pod-autoscaler-sync-period flag, which defaults to 15 seconds. So in the best case there's already a 0–15 second delay before Kubernetes even looks at fresh numbers, stacked on top of however stale metrics-server's own cached data already is.
Cooldowns and Stabilization Windows: Why Autoscalers Wait
Without some kind of pause built in, a naive autoscaler would thrash: add capacity, watch the average metric immediately drop because more instances now share the same load, then remove that capacity again on the very next cycle — flapping back and forth forever.
AWS Auto Scaling handles this with a cooldown period. The default is 300 seconds (5 minutes) for an Auto Scaling Group created through the console; a group created via the CLI or an SDK defaults to 0 seconds unless you set it explicitly. During the cooldown, the group suppresses further scaling activity from simple and step scaling policies so metrics have time to actually settle.
Kubernetes' HPA uses a slightly different mechanism: a stabilization window rather than a single cooldown timer. The default scale-down stabilization window is 300 seconds, while scale-up has effectively no stabilization window by default (0 seconds). Instead of reacting to the newest reading, the controller looks back across the whole trailing window and picks the highest recommended replica count seen anywhere in it — a rolling maximum, not a rolling average.
Both systems land on the same deliberate asymmetry: react almost immediately to more load (protect users first), react slowly to removing capacity (protect against a spike that briefly dips and comes right back). Concretely: traffic spikes at minute 0, HPA scales 4 pods up to 7 within 15–30 seconds; traffic falls back to normal at minute 2; the HPA does not scale back down to 4 immediately — it keeps 7 running until a full 5 minutes of lower readings have passed.
Why Autoscaling Always Reacts Late
Stack up every delay in the pipeline and the total lag becomes obvious:
- Metric collection lag — 5 minutes for EC2 basic monitoring, 1 minute for detailed monitoring, or the HPA's 15-second sync period.
- Decision lag — most autoscalers want several consecutive over-target readings before committing, to avoid reacting to a single noisy sample.
- Provisioning lag — a new EC2 instance has to boot and run its user-data setup; a new pod has to be scheduled, pull its container image, and start.
- Warm-up lag — the application process itself has to start, JVMs have to finish class loading and JIT warm-up, caches need to populate, database connection pools need to ramp up, and the load balancer needs the instance to pass health checks before sending it traffic.
For a Kubernetes pod pulling a moderately sized image and running a JVM app, provisioning plus warm-up alone can easily add 30–90 seconds on top of a control loop that reacted in 15–30 seconds. For an EC2 instance booting from a cold AMI, the total can stretch into minutes.
The practical implication: autoscaling is genuinely good at handling gradual, sustained changes in load. It does not protect a service from a true instant flash spike — a sudden order-of-magnitude traffic jump inside a minute. That needs pre-provisioned headroom, scheduled or predictive scaling for known patterns, or a queue in front of the service that can absorb the burst while capacity catches up.
How Real Systems Combine These Mechanisms
Production setups almost never lean on one lever alone. A typical stack layers several together: reactive horizontal scaling (HPA, or an ASG target-tracking policy) on CPU or a custom metric; a cluster-level autoscaler (Kubernetes' Cluster Autoscaler, or the EC2 Auto Scaling Group underneath a node group) adding actual machines once there's no room left to schedule new pods; the Vertical Pod Autoscaler often run in recommendation-only mode, purely to flag under- or over-sized resource requests, since letting it actually resize live pods means restarts.
Two more pieces show up constantly in real architectures. Scheduled or predictive scaling handles known daily or weekly patterns — an ed-tech platform can scale up before 6pm when students get home from school, so the capacity is already there before the metric-based loop would have reacted at all. Event-driven scaling (KEDA is the common Kubernetes tool for this) scales workers directly off a queue's message count instead of CPU, which reacts earlier — a queue fills up before the workers processing it get visibly busy.