Load Balance Traffic to Private EC2 Instances

Let’s take a look at how we can configure our AWS Application Load Balancer (ALB) to distribute the HTTP traffic across a set of EC2 instances present in the private subnet. As a good security measure, we don’t want our EC2 instances to be directly accessible from the internet. Hence we place them in the private subnet and assign only private IP addresses to them.

2 EC2 instances are present in private subnets across 2 availability zones: us-east-1A and us-east-1B

Let’s take this example, I have 2 EC2 instances present in private subnets across 2 availability zones: us-east-1A and us-east-1B. I want to load balance my web traffic across these two EC2 instances.

Important things to consider -

1. The Internet-facing ALB must be present in the public subnet to receive the incoming web traffic
2. The ALB and EC2 instances must be present in the same availability zones
3. A NAT gateway is required initially to allow the EC2 instances to download the Apache HTTP server. Remove it once the installation is done, typically takes a few minutes to install after launching a fresh EC2 machine. We may put the web server installation command in the EC2 user data section. Use the below commands for Amazon Linux 2 ami.

#!/bin/bash
sudo su
yum update -y
yum install httpd -y
systemctl start httpd
systemctl enable httpd
echo “REQUEST HANDLING BY SERVER in <b>US-East-1A</b>” > /var/www/html/index.html

Follow these steps to create an ALB https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html

After an ALB is created we can access the web application using the DNS name of the ALB.

Access the web app using ALB DNS name

Behind the scene

When we create an ALB, AWS creates the following resources

1. Route53 hosted zone to resolve the ALB hostname to the IP address of the ALB node (ENI). It is NOT visible in the Route53 service.
2. ALB nodes are nothing but Elastic Network Interface (ENI) with public IP addresses. It is visible and we can see the newly created interfaces in the EC2 service.

A simple ‘dig’ of the ALB hostname shows the IP addresses of the two ALB nodes (ENIs)

‘Dig’ command to get the IP addresses of the ALB nodes (ENI)

Routing

1. To access the web application hosted on the EC2 instance, the end user types-in the DNS Name of the ALB. The DNS request goes to Route53 and it resolves the ALB hostname to the IP address of one of the ALB nodes (ENI).
2. The ALB node (ENI) routes the packets to the EC2 instance per the routing table associated with the ALB public subnet. The entry corresponding to 172.31.0.0/16 → Local is used for this purpose. The SG attached to the Load Balancer should allow outbound traffic to the EC2 instance.
3. The web server running on the EC2 instance processes the request and routes the response back to the ALB node (ENI) per the routing table associated with the web server subnet.
4. The ALB node (ENI) sends the web server response to the internet gateway and it goes all the way to the user’s browser.

Common Issue

I’ve followed the steps to create an ALB but my targets still show as unhealthy.

Unhealthy EC2 instances

Root Cause: The web server might not be installed in the EC2 instance. This may be because EC2 machines are launched in the private subnet and don’t have internet access. So when we try to install the web server by putting the installation command in the EC2 user data section it fails.

Resolution: To resolve this create a NAT gateway in the public subnet and modify the routing table of the private subnet to point to the NAT gateway. This is the only step where a NAT gateway is required. It can be removed once the HTTP server is installed on EC2 instances.

Conclusion

We have successfully load balanced our HTTP traffic and when the end user accesses the web application they are routed to one of the healthy EC2 instances.

Request handling by different servers on refreshing the page

References

Amazon ELB for EC2 instances in private subnet in VPC

Amazon ELB in VPC