Load Balancer Issues, Incidents, and Mitigation Strategies #

Performance & Scalability Issues #

Health Checking & Backend Management Issues #

Traffic Distribution & Routing Issues #

Availability & Resilience Issues #

Security Issues #

Configuration & Operational Issues #

Monitoring & Troubleshooting Issues #

Protocol & Traffic Handling Issues #

Advanced Features & Integration Issues #

Infrastructure & Hardware Issues #

Cloud & Distributed Architecture Issues #

Solutions #

Okay, let’s break down how to implement the canonical solution patterns for the Performance & Scalability issues using common load balancer configurations (like Cloud Providers, NGINX, HAProxy).

Remember, load balancer configuration is typically declarative (YAML, text files) rather than procedural code like Java or Rust. However, understanding these concepts is crucial for building robust systems.

Scalable Load Balancing Pattern** #

• Problem Solved: Throughput saturation (Packet Processing, Bandwidth, CPU limits on the LB itself).

• Core Idea: Don’t rely on a single, fixed-size load balancer. Distribute the load balancing function itself horizontally and ensure ample capacity.

• Implementation:

  • Cloud Providers (AWS ELB/ALB/NLB, Azure Load Balancer, GCP Cloud Load Balancing):

    • How: These services are designed for this pattern. They are managed services that automatically scale their underlying capacity based on traffic demand. You don’t manage individual LB instances.
    • Configuration/APIs:
      • Choose the Right Type:
        • AWS: Use Network Load Balancer (NLB) for highest network throughput (millions of PPS), TCP/TLS traffic, and static IPs. Use Application Load Balancer (ALB) for HTTP/S routing, path-based routing, WAF integration, etc. (scales very well but has different capacity characteristics than NLB).
        • Azure: Standard Load Balancer scales automatically. Choose based on L4 vs L7 needs (Application Gateway for L7).
        • GCP: Choose appropriate type (e.g., Global External HTTP(S) LB, Regional Network LB). They scale managedly.
      • Backend Autoscaling: Ensure your backend instances (EC2, VMs, Containers, Functions) are managed by an Autoscaling Group (ASG) or equivalent (Managed Instance Group, VM Scale Set). The load balancer automatically adds/removes targets as the group scales.
      • Monitoring: Monitor the load balancer’s own metrics (e.g., AWS ConsumedLCUs for ALB, ActiveFlowCount for NLB, Azure SNAT Connection Count, GCP LB metrics) to understand usage, though scaling is generally automatic up to service limits. Request limit increases if needed.
      • Headroom: Configure backend ASG policies to scale proactively based on metrics like CPU, request count per instance, or queue depth, ensuring there’s always spare backend capacity before the load balancer needs to throttle.

  • Self-Managed (NGINX, HAProxy):

    • How: You need to run multiple instances of NGINX/HAProxy and distribute traffic to them.
    • Configuration/APIs:
      • Distribution Layer:
        • DNS Round Robin: Simplest but uneven distribution and slow failover.
        • Cloud Network LB: Place an AWS NLB, Azure Standard LB (L4), or GCP Network LB in front of your NGINX/HAProxy instances. The cloud LB handles scaling and distribution to your LB tier.
        • BGP/ECMP (On-Prem/Advanced): Use routing protocols to advertise the same IP from multiple LB instances, letting routers distribute traffic (requires network team involvement).
      • High Availability (HA): Use tools like keepalived (VRRP) for active/passive or active/active setups if not using a higher-level distribution method like a cloud NLB.
      • Instance Scaling: Run your NGINX/HAProxy instances within an Autoscaling Group (if in the cloud) so the LB tier itself can scale.
      • Configuration Example (Conceptual - using Cloud NLB):

        [ Internet ] -> [ AWS NLB / Azure LB / GCP Network LB ] -> [ ASG of NGINX/HAProxy Instances ] -> [ Backend App Servers ]
        

      • NGINX/HAProxy Config: The individual configs don’t change much for this pattern, it’s about deploying multiple copies effectively.

Connection Management Pattern** #

• Problem Solved: Connection exhaustion (running out of connection table entries, ephemeral ports, or file descriptors).

• Core Idea: Tune operating system and load balancer limits, manage timeouts effectively, and potentially limit connections per client.

• Implementation:

  • Cloud Providers:

    • How: Limits are often high but exist. Focus on timeouts and choosing the right LB type.
    • Configuration/APIs:
      • Check Service Quotas: Understand the documented connection limits for your chosen LB type (e.g., AWS ALB/NLB limits, Azure LB limits).
      • Idle Timeouts: Configure appropriate idle timeouts. Longer timeouts hold connections open; shorter timeouts reclaim resources faster but might break long-polling/WebSockets. (e.g., AWS ALB idle_timeout.timeout_seconds, Azure LB Idle Timeout). NLBs are generally better for very long-lived connections.
      • Client IP Preservation: Ensure backend servers see the actual client IP (using Proxy Protocol or X-Forwarded-For) if they need to manage connections based on client.

  • Self-Managed (NGINX, HAProxy):

    • How: Direct control over OS and LB parameters.
    • Configuration/APIs:
      • Operating System Tuning (sysctl.conf, limits.conf):
        • net.core.somaxconn: Max backlog of pending connections (e.g., 65535).
        • net.ipv4.tcp_max_syn_backlog: Max SYN queue size (e.g., 65535).
        • fs.file-max: System-wide max file descriptors (e.g., 200000 or more).
        • nofile (via ulimit or /etc/security/limits.conf or systemd unit file): Max file descriptors per process (e.g., 65535 or 1048576). Connections consume file descriptors.
      • NGINX (nginx.conf):
        • worker_processes auto; (Usually set to number of CPU cores).
        • worker_rlimit_nofile 65535; (Set worker process file descriptor limit).
        • events { worker_connections 16384; } (Max connections per worker. Total ≈ worker_processes * worker_connections, limited by nofile).
        • keepalive_timeout 65; (Adjust idle connection timeout).
        • keepalive_requests 1000; (Max requests per keepalive connection).
        • client_header_timeout, client_body_timeout, send_timeout: Tune client timeouts.
        • Client Limiting:

          http {
              limit_conn_zone $binary_remote_addr zone=addr:10m; # Define zone based on client IP
              server {
                  ...
                  location / {
                      limit_conn addr 10; # Limit to 10 concurrent connections per IP
                  }
              }
          }
          

      • HAProxy (haproxy.cfg):
        • global:
          • maxconn 100000 (Global max concurrent connections).
          • tune.maxaccept 100 (How many connections to accept at once).
          • ulimit-n 200050 (Set nofile limit directly).
        • defaults or frontend:
          • maxconn 50000 (Per-frontend limit).
          • timeout client 30s (Adjust client idle timeout).
          • timeout server 30s (Adjust server idle timeout).
          • timeout connect 5s (Backend connection attempt timeout).
        • Client Limiting:

          backend app
              stick-table type ip size 1m expire 30s store conn_cur # Store current connections per IP
              tcp-request connection track-sc1 src
              tcp-request connection reject if { sc1_conn_cur ge 10 } # Reject if IP has >= 10 connections
          

Optimized TLS Pattern** #

• Problem Solved: SSL/TLS processing bottlenecks (high CPU usage during handshakes).

• Core Idea: Use hardware acceleration where possible, optimize cipher selection, and enable session reuse mechanisms.

• Implementation:

  • Cloud Providers:

    • How: Leverage built-in optimizations and hardware acceleration.
    • Configuration/APIs:
      • Hardware Acceleration: Usually handled automatically by the managed service.
      • Security Policies: Select predefined TLS Security Policies (e.g., AWS ELBSecurityPolicy-TLS-1-2-Ext-2018-06, Azure App Gateway SSL Policies) that balance security and performance. Avoid legacy policies unless absolutely required. Check cipher suites included in the policy.
      • Session Resumption: Typically enabled by default (Session IDs/Tickets). Verify in documentation if specific controls exist.
      • Protocols: Ensure modern protocols like TLS 1.2 and TLS 1.3 are enabled in the policy.

  • Self-Managed (NGINX, HAProxy):

    • How: Explicit configuration of ciphers, protocols, session caching, and potentially linking against hardware acceleration libraries.
    • Configuration/APIs:
      • Hardware Acceleration: If your hardware supports it (e.g., Intel QAT, specialized cards), you may need specific builds of NGINX/HAProxy or OpenSSL compiled with engine support. Consult hardware/software documentation.
      • NGINX (nginx.conf - within server block listen ... ssl):

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;
        # Example: Modern cipher suite prioritizing AEAD ciphers
        ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
        ssl_ecdh_curve prime256v1:secp384r1; # Efficient curves
        
        # Session Cache (Session IDs)
        ssl_session_cache shared:SSL:10m; # 10MB cache (~40,000 sessions)
        ssl_session_timeout 10m;
        
        # Session Tickets (Alternative/complementary, keys need rotation)
        ssl_session_tickets on;
        # ssl_session_ticket_key /path/to/ticket.key; # Needs secure generation and rotation
        

      • HAProxy (haproxy.cfg - within bind line):

        # Global or frontend/defaults section
        tune.ssl.default-dh-param 2048
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
        ssl-default-bind-options ssl-min-ver TLSv1.2 # Enforce minimum TLS version
        
        frontend myfrontend
           bind *:443 ssl crt /path/to/cert.pem alpn h2,http/1.1 # Add ciphers/options here if not default
           # Session resumption enabled by default, session cache automatic
        

Coordinated Scaling Pattern** #

• Problem Solved: Uneven scaling across tiers in a multi-layer LB architecture (e.g., external LB scales but internal LB doesn’t, creating a bottleneck).

• Core Idea: Link or coordinate the scaling actions across different tiers based on relevant, potentially cross-tier, metrics.

• Implementation: (This is more architectural and automation-focused than specific LB config)

  • Cloud Providers:

    • How: Use cloud monitoring and autoscaling features, potentially with custom logic.
    • Configuration/APIs:
      • Tiered Monitoring: Set up monitoring (CloudWatch, Azure Monitor, Google Cloud Monitoring) for key metrics at each tier (e.g., Edge LB connections/throughput, Internal LB active connections/latency, App Server CPU/Memory/Queue).
      • Independent Scaling: Ensure each tier (Edge LB Target Group ASG, Internal LB Target Group ASG, App Tier ASG) has its own autoscaling policies based on its most relevant metrics.
      • Cross-Tier Triggers (Advanced):
        • Use custom metrics or Lambda/Functions triggered by alarms. Example: If Edge LB ActiveFlowCount > Threshold AND Internal LB TargetResponseTime > Threshold, trigger scaling of the Internal LB’s ASG or even the App Tier’s ASG.
        • AWS: CloudWatch Metric Math can create combined metrics for alarms. Step Functions or Lambda can implement more complex scaling logic.
        • Azure: Azure Monitor action groups can trigger Azure Functions or Logic Apps.
        • GCP: Cloud Monitoring alerting can trigger Cloud Functions or Pub/Sub.
      • Capacity Headroom: Over-provision slightly or set scaling thresholds lower at downstream tiers so they can react before becoming bottlenecks when an upstream tier scales up traffic.

  • Self-Managed (NGINX, HAProxy) + Orchestration (Kubernetes, etc.):

    • How: Combine LB metrics with orchestration platform scaling.
    • Configuration/APIs:
      • Metrics Exposure: Ensure your LBs expose detailed metrics (e.g., using nginx-prometheus-exporter, HAProxy’s built-in Prometheus endpoint, or stats socket).
      • Monitoring System: Scrape metrics using Prometheus or similar.
      • Kubernetes HPA: Use Horizontal Pod Autoscalers.
        • Scale LB deployments based on their own CPU/Memory or custom metrics (e.g., active connections scraped by Prometheus Adapter).
        • Scale backend application deployments based on their CPU/Memory, custom metrics (e.g., RPS per pod), or external metrics (like queue length).
      • Custom Controllers/Operators: For complex coordination logic (similar to the cloud Lambda approach), a custom Kubernetes operator might observe metrics across different deployments/services and adjust HPA targets or replica counts directly.
      • Service Mesh: If using a service mesh (Istio, Linkerd), leverage its observability and potentially traffic splitting capabilities to manage scaling and load distribution internally. The external LB still needs to scale appropriately.