A site that handles a hundred concurrent visitors can collapse when a thousand arrive, perhaps because an article was shared or a promotional campaign started. Keeping enough servers running for the maximum peak at all times is expensive; keeping too few is risky. Load balancing and autoscaling solve this dilemma by adapting resources to real traffic.
Two ways to grow
Vertical scaling
You replace the server with a more powerful one: more CPU, more memory. It is simple, requires no changes to the application and works well up to a point. There are three limits: there is still a largest possible machine, changing size often requires a reboot, and a single server remains a single point of failure.
Horizontal scaling
You add more identical servers and split the work among them. It has no theoretical limit, lets you survive the failure of one machine and can be done without interruptions. In exchange it requires the application to be designed to run as several copies at once, and a component that distributes traffic: the load balancer.
The load balancer
The load balancer receives all requests and distributes them among the available servers. The most common distribution criteria are:
- round robin: requests go to each server in turn;
- least connections: the request goes to the least busy server at that moment;
- hash of the address or a header: the same client always lands on the same server.
Load balancers work at two different levels. Layer 4 balancers dispatch TCP or UDP connections without looking at their content: they are fast and generic. Layer 7 balancers read HTTP requests and can route by domain, path or headers, handle certificates and add information.
Health checks
A balancer without health checks keeps sending requests to a stuck server. With health checks, it periodically queries an application endpoint, for example /health, and stops using servers that don't respond correctly, putting them back into rotation when they are healthy again.
A good health check verifies that the application can really serve requests, but without depending on too many external services: if the check fails every time a secondary service slows down, the balancer risks excluding all servers at once.
Stateless applications
Horizontal scaling only works well if any server can handle any request. This means the application must be stateless:
- user sessions must be stored in a shared store, such as Redis or the database, not in a single server's memory;
- uploaded files go into a common object storage, not onto the local disk;
- scheduled jobs must be run by a single server or coordinated, otherwise they run as many times as there are servers.
There is the shortcut of sticky sessions, where the balancer always sends the same user to the same server. It works, but distributes load less evenly and loses sessions when a server is removed. It is best regarded as a temporary solution.
Autoscaling
Autoscaling adds and removes servers automatically based on a metric. It is configured with a few parameters:
| Parameter | Example | Meaning |
|---|---|---|
| Minimum | 2 | Number of instances always running, even at night |
| Maximum | 10 | Ceiling beyond which it won't grow, to contain costs |
| Metric | Average CPU at 60% | Value the system tries to maintain |
| Cooldown | 5 minutes | Minimum wait between two successive actions |
A minimum of two instances ensures that a failure doesn't stop the service. The maximum protects against a mistake or an attack that would make spending grow without limit.
Choosing the right metric
CPU is the most used metric, but not always the most suitable. An application that spends most of its time waiting for the database can be slow with low CPU. Valid alternatives are requests per instance, response time or, for background jobs, the length of the queue to process.
Reaction times
A new instance isn't ready instantly: it must boot, load the application and pass health checks. If that takes four minutes and the spike arrives in thirty seconds, autoscaling steps in too late. Helpful measures include prebuilt images with everything installed, applications that start quickly and, for predictable events, a scheduled increase in instances before the traffic arrives.
The bottleneck moves
Adding application servers is useless if the limit lies elsewhere. Often, when instances double, the database receives twice the connections and becomes the problem itself. Before relying on autoscaling it pays to check with a load test where the system really gives way: database, cache, external services with request limits. In my work this test almost always reveals an unexpected bottleneck, and it is much better to discover it in a test than during the peak.
In short
- Vertical scaling is simple but limited; horizontal scaling requires applications designed for multiple copies.
- The load balancer distributes requests and, thanks to health checks, excludes servers that aren't working.
- Sessions, files and scheduled jobs must be handled outside the single server.
- Autoscaling is configured with a minimum, maximum, metric and cooldown, chosen with care.
- A load test shows where the real bottleneck lies before the peak arrives.
