Load Balancer

A component that distributes incoming network requests across multiple backend servers so that no single server is overloaded. It provides a single entry point, improves throughput, and increases availability by routing around failed servers.

• Scalability: add servers to handle more load.
• High availability: reroute traffic away from unhealthy servers, enabling zero-downtime deploys and maintenance.

Requests are handed to servers in a fixed rotation: server 1, 2, 3, 1, 2, 3, … It is simple and works well when servers have equal capacity and requests are roughly equal in cost.

A periodic probe (e.g. an HTTP GET /healthz or a TCP connect) the balancer sends to each backend. If a server fails repeatedly it is removed from the pool; when it passes again it is re-added.

No. The client connects to one stable address (the balancer's VIP or DNS name). The number and identity of backends is hidden and can change at any time.

Hardware: dedicated appliances like F5 BIG-IP or Citrix. Software: HAProxy, NGINX, Envoy, or cloud services like AWS ELB/ALB/NLB and GCP Cloud Load Balancing. Software/cloud balancers dominate modern deployments for cost and flexibility.

A stateless server keeps no per-user data in local memory between requests; all shared state lives in a database or cache. This lets any server handle any request, so the balancer can freely distribute traffic and you avoid sticky sessions.

L4 operates on TCP/UDP: it forwards packets based on IP address and port without reading the payload. It is fast and protocol-agnostic. L7 understands the application protocol (usually HTTP), so it can route by URL path, headers, cookies, or method; terminate TLS; and rewrite/aggregate requests. L7 is more flexible but does more work per request.

When request durations vary widely (e.g. some requests hold connections for seconds, others milliseconds). Round-robin can pile long requests onto one server; least-connections routes new work to whichever server currently has the fewest active connections, balancing actual load better.

Session affinity pins a client to a specific backend (usually via a cookie or source-IP hash) so in-memory session state is found on subsequent requests. Drawbacks: uneven load distribution, lost sessions when a server dies, and harder scaling. The preferred alternative is stateless servers with sessions stored in a shared store like Redis.

Active connections to the dead server break; the client (or the L7 balancer itself, if idempotent) may retry, and the request is routed to a healthy server. Health checks then mark the server down so no new traffic is sent. For safe retries the request must be idempotent, or you risk double-processing.

The L7 balancer decrypts HTTPS at the edge and forwards plaintext (or re-encrypted) traffic to backends. Benefits: centralized certificate management, offloading crypto CPU from app servers, and the ability to inspect/route on HTTP content. Trade-off: traffic between LB and backend should still be encrypted in zero-trust networks (TLS re-encryption / mTLS).

When you remove a server (for a deploy or scale-in), the balancer stops sending it new connections but allows existing in-flight requests to complete for a configured timeout before the server shuts down. This avoids dropping active users' requests during deployments.

Active: the balancer proactively sends probe requests on a schedule. Passive (outlier detection): the balancer watches real traffic and ejects a backend that returns errors or high latency, even if its active probe still passes. Using both catches more failure modes.

Yes — DNS can return multiple A records or rotate them (round-robin DNS), and GeoDNS can return region-specific IPs. Limits: DNS caching/TTLs mean changes propagate slowly, clients may cache a dead IP, and there is no health awareness or fine-grained balancing. DNS is best for coarse, global steering, with real balancers underneath.

Each backend gets a weight proportional to its capacity, and traffic is distributed accordingly. Useful for heterogeneous fleets (mixed instance sizes), canary releases (send 5% to a new version), or gradually shifting traffic during migrations.

They overlap. A reverse proxy accepts client requests on behalf of servers (TLS, caching, compression, routing). A load balancer's defining job is distributing traffic across multiple backends. Most L7 products (NGINX, Envoy, HAProxy) are both: reverse proxies that also load balance.

Run at least two balancer nodes. Options: active-passive with a floating IP via VRRP/keepalived (the standby takes the VIP on failure); active-active behind anycast or DNS so multiple balancers serve simultaneously; or use a managed cloud LB that is internally redundant across availability zones.

NLB is Layer 4 (TCP/UDP), ultra-low latency, static IPs, millions of connections. ALB is Layer 7 (HTTP/HTTPS) with path/host routing, WebSockets, and target groups. Classic ELB is the older generation supporting basic L4/L7. Choose NLB for raw TCP throughput, ALB for HTTP-aware routing.

You deploy the new version to a subset of servers (or a parallel "green" pool), health-check them, then shift traffic gradually (weighted routing / blue-green / canary). Old servers are drained and removed. Because the balancer only routes to healthy nodes, users never hit a server mid-restart.

With hash(key) % N, changing N remaps almost every key, destroying cache locality. Consistent hashing maps servers and keys onto a ring; a key goes to the next server clockwise. Adding/removing a server only remaps keys in that server's arc (~1/N of keys). Virtual nodes (multiple ring positions per server) smooth out imbalance. This is essential when backends hold caches or session affinity so a topology change doesn't invalidate everything.

Layer it: (1) GeoDNS or anycast directs users to the nearest healthy region; (2) a regional L4 balancer spreads across clusters/AZs with static IPs; (3) an L7 balancer / service mesh does path routing, retries, circuit breaking, and canaries within the region. Add global health checks to fail a whole region out via DNS, cross-region failover with capacity headroom, and latency-based routing. Watch DNS TTLs and connection reuse for failover speed.

The same IP is advertised via BGP from many locations; the network routes each client to the topologically nearest instance. Pros: automatic geo-steering, DDoS absorption, fast failover (withdraw a route). Cons: routing changes can move a client mid-session (bad for long-lived TCP unless handled), and you have limited control over exactly which PoP a user hits. Common for DNS, CDNs, and DDoS scrubbing.

When a backend or region recovers (or many clients disconnect at once), all clients reconnect simultaneously, spiking load and possibly re-crashing the node. Mitigations: randomized (jittered) exponential backoff on clients, gradual re-admission / slow start on the balancer, connection rate limiting, and load shedding at the edge until caches warm.

Balancing happens at connection setup, not per message, so a naive L4 balancer can leave load skewed as connections persist. Use least-connections or connection-count-aware algorithms, enable graceful draining that closes idle connections, and for gRPC (which multiplexes many requests over one HTTP/2 connection) prefer an L7/mesh proxy that balances per-request, or use client-side load balancing with service discovery. Periodically force reconnection to rebalance.

The client fetches the list of healthy backends from service discovery (e.g. Consul, Eureka, xDS) and picks one itself, removing the network hop and central bottleneck of a dedicated LB. Preferable inside a mesh/microservices environment for low latency and per-request balancing (e.g. gRPC). Trade-offs: clients must embed balancing logic, handle discovery freshness, and you lose a central control/observability point (mitigated by a sidecar like Envoy).

Combine several techniques: passive outlier detection to eject high-latency/error backends; per-request timeouts and hedged/retry requests to fast peers; circuit breakers to stop hammering a failing node; concurrency limits / bulkheads so one slow dependency can't exhaust all connections; and least-response-time balancing so traffic naturally avoids slow nodes. The "power of two choices" (pick 2 random backends, send to the less loaded) is a cheap, effective algorithm here.

Instead of tracking global state or strict least-connections, pick two backends at random and route to whichever has fewer active connections. This gets nearly the load-smoothing quality of full least-connections with O(1) work and no global coordination, and it dramatically reduces the maximum load compared to pure random. It scales well across distributed balancers that each have only partial load information.

Autoscalers add/remove instances; the balancer must register new ones (with slow start so cold instances aren't hammered) and drain removed ones. Pitfalls: registering an instance before it is truly ready (use readiness vs liveness probes), overly aggressive health checks flapping instances in and out, scale-in terminating an instance mid-request without draining, and health-check storms adding load. A death spiral can occur if health checks fail under load, removing capacity, which increases load on the rest.

Equal request counts don't mean equal work. Causes: heterogeneous request cost (some endpoints are expensive), heterogeneous hardware, "heavy hitter" clients pinned by session affinity or consistent hashing, background jobs on some nodes, GC pauses, or hot cache keys concentrating on certain shards. Fix by balancing on a better signal (least-response-time or CPU-aware weighting), sharding hot keys, or splitting expensive endpoints into their own pool.

You generally don't put a naive LB in front of a primary DB. Instead: route writes to the leader and reads to replicas (read/write splitting, often in the driver or a proxy like ProxySQL/PgBouncer), use connection pooling to bound connections, and shard data so each shard has its own leader. For failover, a coordinator (e.g. Patroni, Orchestrator) promotes a replica and updates the routing target. Consistency requirements dictate whether reads can go to lagging replicas.