Load balancing: how to distribute traffic across servers

A load balancer distributes incoming requests across several application servers, monitors their health and stops sending traffic to broken ones. You need one when a single server lacks capacity or the site must not go down because one machine fails. For balancing to work, the application must not keep state on a specific server.

Here is how load balancers work, which algorithms exist, how to prepare the application and how to keep the balancer itself from becoming a single point of failure.

Why do you need a load balancer?

A load balancer solves two problems: scaling and fault tolerance. Instead of one ever-larger server you can run several ordinary ones and add more as you grow, and if one breaks, users do not notice.

What balancing gives you:

• horizontal scaling — load is shared across servers;
• fault tolerance — a broken server is removed automatically;
• zero-downtime releases — servers are updated one at a time; see zero-downtime deployments;
• a single entry point — TLS certificates, compression and limits are configured in one place.

What kinds and algorithms of balancing are there?

Load balancers work at the transport layer (L4), distributing TCP connections without looking inside, or at the application layer (L7), understanding HTTP and routing by URL, headers and cookies. Web applications usually use L7.

Algorithm	How it works	When it fits
Round robin	Each server in turn	Identical servers and requests
Weighted round robin	More requests to more powerful servers	Servers of different capacity
Least connections	To the server with the fewest connections	Requests of varying duration
Hash by IP or key	One client always goes to one server	When affinity is needed, e.g. for caching

Photo: Unsplash

How do health checks work?

The balancer regularly polls each server with a special request — a health check. If a server fails to respond or returns errors several times in a row, the balancer takes it out of rotation and brings it back once it is healthy again.

upstream app {
    least_conn;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080 backup;
}

server {
    listen 443 ssl;
    http2 on;
    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_502 http_503;
    }
}

A good health check verifies not just that the process is alive but that the application can work: it has database and cache connections. It must also be fast and not load the system.

How do you prepare an application for several servers?

The application must be stateless: any user request can land on any server, so nothing important can live in a specific machine's memory or disk. Sessions, cache, uploaded files and scheduled tasks move to shared stores.

1. Sessions — in Redis or the database, not in server files.
2. Uploaded files — in object storage; see file uploads and storage.
3. Cache and locks — in a shared Redis; see Redis in a web project.
4. Scheduled tasks — run on one server or with a distributed lock, or they will execute several times.
5. Logs — collected centrally rather than kept on each server.
6. The client's real IP — passed by the balancer in a header and read correctly by the application.

How do you keep the balancer from being a single point of failure?

If there is only one balancer, its failure takes the whole site down. So it is made redundant too: use a cloud load balancer the provider keeps fault-tolerant, or run two of your own sharing a floating IP address.

• a cloud load balancer — the simplest option, with redundancy handled by the provider;
• two balancers with address failover if one dies;
• DNS with several addresses and health checks — to spread traffic across regions;
• monitoring the balancer itself: 5xx errors, response time, number of healthy servers.

On choosing infrastructure, see VPS, cloud or managed services and do you need Kubernetes.

In short

• A load balancer spreads requests across servers and removes broken ones.
• Web apps usually use L7 balancing with least connections or round robin.
• Health checks should verify the application is ready, not just that the process runs.
• The application must be stateless: sessions, files and cache live in shared stores.
• Make the balancer redundant too, or it becomes a single point of failure.