Load balancing distributes traffic or connections across multiple backend servers. Use a load balancer when one application server has become an unacceptable capacity, availability, deployment, or maintenance boundary. A load balancer can route new work away from unhealthy backends, but it does not automatically make the database, storage layer, DNS, deployment process, or load-balancer service itself highly available.
The application must also be ready to run on multiple nodes. Sessions, uploads, jobs, health checks, timeouts, and remaining capacity after a failure determine whether the design works in production.
Load balancing is justified by a specific constraint
Add a load balancer when at least one of these requirements is real:
- one server failure would cause unacceptable downtime
- one server cannot handle expected peak traffic
- deployments or maintenance should not interrupt all users
- application nodes can process work in parallel
- different routes or services need separate backend pools
- internal services need a stable private entry point
Do not add one because a multi-server architecture appears more advanced. It will not fix:
- inefficient queries
- a memory leak
- a full or slow disk
- an overloaded database
- an unavailable external API
- local sessions or files that prevent traffic from moving between nodes
Use Cloud Server Performance Bottlenecks before scaling the wrong layer.
Use a load-balancing decision framework
Before implementation, answer these questions:
- Goal: capacity, availability, safer deployment, maintenance, or routing?
- Protocol: HTTP/HTTPS, raw TCP, UDP, TLS passthrough, or another protocol?
- State: can any backend process the next request?
- Health: what proves a backend is ready for real traffic?
- Failure capacity: can the remaining servers carry the workload after one fails?
- Traffic behavior: short requests, long connections, streaming, WebSockets, or background jobs?
- Security: where is TLS terminated, and which backend ports are reachable?
- Operations: how will nodes be added, drained, deployed, monitored, and removed?
A load balancer is useful only when the complete application path supports these decisions.
Vertical scaling and horizontal scaling solve different problems
Vertical scaling increases the resources of one server. Horizontal scaling adds servers and distributes work between them.
| Situation | Better first move |
|---|---|
| CPU or memory is the measured limit | Optimize or resize the VM |
| One-node failure is unacceptable | Use multiple backends and load balancing |
| Database is the bottleneck | Tune or scale the database first |
| Sessions and files remain local | Externalize state before scaling out |
| Deployments interrupt users | Add redundant backends and draining |
| Traffic is low and requirements are simple | Keep one VM until complexity is justified |
Read Horizontal vs Vertical Scaling for the broader decision.
How a load balancer handles traffic
A basic request path is:
Client ↓ DNS ↓ Load balancer listener ↓ routing decision Healthy backend pool ↓ Database and internal services
The load-balancing design includes:
- a frontend address and listener
- protocol and port
- one or more backend pools
- routing rules or an algorithm
- health checks
- connection and idle timeouts
- TLS behavior
- source-IP forwarding behavior
- draining and backend lifecycle controls
The backend application must trust forwarded client information only from approved proxy sources. For HTTP systems, this may include forwarded address, scheme, host, and request identifiers. Incorrect proxy trust can produce wrong redirects, insecure URL generation, or spoofed client-address logs.
L4 vs L7 load balancing
Layer 4 load balancing uses transport-level information such as IP addresses, ports, and protocols. Layer 7 load balancing understands application-layer requests, commonly HTTP and HTTPS.
| Capability | Layer 4 | Layer 7 |
|---|---|---|
| Typical protocols | TCP, UDP, TLS | HTTP, HTTPS |
| Routing basis | Address, port, connection | Hostname, path, header, cookie, method |
| Application awareness | Limited | High |
| TLS model | Passthrough or termination depending on service | Commonly termination and re-encryption |
| Good fit | Databases, game traffic, custom protocols, simple connection forwarding | Websites, APIs, microservices, redirects, route-based pools |
AWS documents Application Load Balancers as Layer 7 and Network Load Balancers as Layer 4. Google Cloud similarly distinguishes Layer 4 decisions based on network and transport information from Layer 7 decisions based on HTTP attributes.
Choose Layer 4 when the load balancer should distribute connections without interpreting the application request. Choose Layer 7 when routing or policy depends on HTTP behavior.
External and internal load balancers serve different boundaries
An external load balancer accepts traffic from internet clients. An internal load balancer provides a stable entry point for services inside a private network.
Internet ↓ External load balancer ↓ Web or API nodes ↓ private network Internal service load balancer ↓ Worker or service nodes
An internal load balancer can simplify service discovery and backend replacement without exposing the service publicly. It still requires firewall rules, authentication, and capacity planning.
Routing algorithms must match workload behavior
Common algorithms include:
- Round robin: distributes new requests or connections in sequence.
- Weighted round robin: sends a larger share to selected backends.
- Least connections: favors the backend with fewer active connections.
- Hash-based routing: selects a backend from a stable client or request attribute.
- Random or power-of-two choices: selects among available backends with low coordination overhead.
Round robin works well when backends and request durations are similar. Least connections may be more suitable for variable or long-lived connections. Weights are useful during migrations, canary releases, or mixed backend sizes.
No algorithm fixes an overloaded shared database or an application that cannot process requests on arbitrary nodes.
Health checks should measure readiness
Health checks determine whether a backend remains eligible for new traffic.
A port-open check proves only that something accepted a connection. A readiness check should prove that the application can serve the intended workload.
A useful health endpoint should be:
- fast
- deterministic
- inexpensive
- safe to call frequently
- strict enough to detect an unusable node
- independent of optional dependencies
Separate these concepts:
| Check | Question |
|---|---|
| Liveness | Is the process running or recoverable? |
| Readiness | Can this instance accept new production traffic? |
| Dependency health | Are required downstream services available? |
| Synthetic workflow | Can a representative user action complete? |
Do not put every downstream dependency into a shallow readiness endpoint. A temporary optional-service failure should not necessarily remove the entire application fleet. Conversely, a check that returns success while the application cannot reach its required database may be too weak.
AWS health-check documentation describes configurable intervals, timeouts, healthy thresholds, and unhealthy thresholds. Tune them according to failure-detection needs and normal latency rather than copying one universal value.
Decide what happens when every backend is unhealthy
Load-balancer behavior when all targets fail health checks differs by platform and configuration. A system may stop routing, continue routing to unhealthy targets, return an error, or follow another fail-open or fail-closed policy.
Verify this behavior before production. The choice affects:
- whether users reach a degraded but partially working service
- whether traffic amplifies a failure
- whether a maintenance event becomes a full outage
- what monitoring and incident response should expect
A health-check configuration is incomplete until the team understands the all-backends-unhealthy case.
Connection draining protects in-flight work
When a backend is removed for deployment, maintenance, or scaling, the load balancer should stop sending it new work while existing requests or connections finish.
AWS calls this connection draining through a deregistration delay. Its documentation recommends deregistering the target and allowing in-flight requests to complete before stopping the application.
A safe backend removal sequence is:
- Mark the node unavailable for new traffic.
- Wait for routing configuration to propagate.
- Allow active requests and connections to complete.
- Stop workers or application processes safely.
- Deploy, maintain, or terminate the node.
- Return it only after readiness succeeds.
The draining period should reflect real request and connection duration. A short API request and a long file upload, stream, or WebSocket connection need different handling.
Slow start can protect a newly added backend
A new node may need time to warm caches, establish connection pools, load application data, or complete just-in-time compilation.
Sending a full traffic share immediately can make the new backend fail its first health checks or overload dependencies. Where the platform supports it, gradual traffic ramp-up can reduce this risk.
Otherwise, perform warm-up before readiness or introduce the backend with temporary weights.
Timeouts should align across the path
A request may pass through:
- client
- CDN or edge proxy
- load balancer
- reverse proxy
- application server
- database or external API
Each layer may have connection, idle, request, and response timeouts. Misaligned values create confusing failures such as clients waiting longer than the load balancer, or backends continuing expensive work after the client has already received an error.
Document:
- connection-establishment timeout
- idle timeout
- backend response timeout
- upload and streaming behavior
- application cancellation behavior
- retry policy
Retries can improve resilience for safe, transient failures, but retries at several layers can multiply traffic. Do not automatically retry non-idempotent actions such as charges or state-changing requests without an idempotency design.
Stateless application nodes scale more safely
Load balancing works best when any healthy backend can process the next request.
| Local dependency | Better shared pattern |
|---|---|
| In-memory session | Shared session store or suitable token design |
| User uploads | Object storage or shared storage |
| Durable application files | External authoritative storage |
| Background jobs | Shared queue with retry and ownership |
| Required cache | Shared cache with defined loss behavior |
| Local database | Separate database service or VM |
Sticky sessions can preserve compatibility for legacy applications, but they keep users dependent on a selected backend and can produce uneven traffic.
Use stickiness as a deliberate trade-off, not as evidence that the application is highly available. Read Stateful vs Stateless Applications.
TLS design changes routing and trust boundaries
Common TLS models include:
| Model | Traffic path |
|---|---|
| Termination | HTTPS to load balancer, HTTP to protected backend |
| Re-encryption | HTTPS to load balancer, HTTPS to backend |
| Passthrough | Encrypted connection forwarded to backend |
Termination centralizes public certificate management and enables Layer 7 request inspection. Re-encryption protects the backend path. Passthrough keeps termination at the application but limits HTTP-aware routing.
When TLS terminates before the application, configure the application to recognize the original scheme safely through trusted proxy information. Restrict backend ports so clients cannot bypass the intended TLS and routing boundary.
Load-balancer failover requires spare capacity
Failover means new traffic stops going to an unhealthy backend and is directed to healthy ones.
The remaining fleet must have enough capacity to absorb the failed node. Three servers running near full capacity during normal traffic may still form a fragile design if losing one overloads the others.
Plan for:
- one-backend failure
- maintenance while another backend is unavailable
- traffic spikes during recovery
- cache warming on remaining nodes
- database connection redistribution
High availability is a capacity decision as well as a routing decision.
A load balancer does not remove every single point of failure
A multi-backend application can still fail because of:
- load-balancer service or configuration failure
- DNS problems
- one primary database
- shared storage failure
- common secrets or configuration errors
- a bad deployment sent to every backend
- unavailable external dependencies
- insufficient capacity after failover
Load balancing improves the application tier. High Availability vs Disaster Recovery explains the wider system decision.
Backups solve data recovery, not live traffic routing.
Database pressure can increase after scaling out
Adding application nodes commonly increases:
- database connections
- concurrent queries
- lock contention
- cache misses
- queue consumers
- outbound API calls
Before adding backends, review connection pooling, query latency, storage performance, rate limits, and database failover requirements.
A faster application tier can move the bottleneck into the data layer.
Deployments should use load-balancer lifecycle controls
A rolling deployment can update one backend at a time:
- Drain one node.
- Deploy the new version.
- Wait for readiness.
- Restore traffic gradually.
- Continue with the next node.
Old and new versions may run together. Database schemas, session formats, queue messages, cache keys, and API contracts must remain compatible.
Load balancing also supports canary behavior through weighted pools where available, but the team still needs rollback criteria and observability.
Monitor the complete traffic path
Useful load-balancer and backend signals include:
- request or connection rate
- p50, p95, and p99 latency
- frontend and backend errors
- healthy and unhealthy backend count
- health-check failure reasons
- active connections
- connection resets
- timeout responses
- backend saturation
- traffic distribution by node
- draining duration
Correlate these signals with deployments and application logs. A healthy load balancer can still route traffic to an application that returns logical errors, while an unhealthy target may reflect a firewall, TLS, timeout, or application problem.
A practical cloud architecture
Internet users ↓ Public load balancer ↓ Private application VM 1 Private application VM 2 ↓ Private database and services ↓ Backups and monitoring
Recommended boundaries:
- expose the intended public entry point
- keep backends on private networking where practical
- restrict backend firewall rules to approved sources
- externalize sessions and uploads
- size for one-node failure
- drain nodes before maintenance
- monitor user impact and backend health separately
How this applies on Raff
A Raff architecture may combine multiple Cloud Servers, Raff VPC, security controls, and the current Load Balancers offering where it matches the required protocol and routing model.
Use the live product and pricing pages to verify current availability, supported protocols, TLS behavior, health checks, limits, regions, and billing before implementation. Do not rely on historical launch or pricing statements in an evergreen architecture guide.
Load-balancing checklist
Architecture
- The reason for multiple backends is documented.
- The application can process requests on arbitrary healthy nodes.
- Sessions, files, and jobs are externalized where required.
- Database and dependency headroom is measured.
Routing
- L4 or L7 is selected according to protocol needs.
- The algorithm matches connection and request behavior.
- Client and proxy headers are trusted only from approved sources.
- TLS termination and backend encryption are defined.
Health and lifecycle
- Readiness checks represent real serving ability.
- All-backends-unhealthy behavior is understood.
- Draining is tested with real request durations.
- New-node warm-up is planned.
Reliability
- Remaining nodes can absorb one-backend failure.
- Timeouts and retries are aligned across the path.
- The load balancer, DNS, database, and storage failure domains are reviewed.
- Monitoring distinguishes routing health from application health.
:::cluster
Conclusion
Load balancing is appropriate when one application server should no longer define capacity, availability, deployment safety, or maintenance behavior.
Choose Layer 4 or Layer 7 according to the protocol and routing requirement. Build meaningful readiness checks, externalize state, align timeouts, drain nodes safely, and preserve enough capacity for backend failure.
A load balancer improves traffic distribution. Reliable architecture still depends on the health, state, capacity, and recovery design of every service behind it.