Amazon’s Elastic Load Balancers have been the delicious fruit topping on our application deployments since Cloudamatic’s inception. They’re a flexible and cost-effective component for all sorts of purposes:
- Distributing web application traffic across multiple backend nodes for horizontal scaling
- Providing a consistent endpoint when a node fails and must be replaced
- Providing an https front-end for web applications which support encryption poorly (or not at all)
- Serving as a temporary public RDP gateway for hosts in private subnets
- Request logging for backend applications which suffer a performance penalty with full logging enabled (e.g. ArcGIS)
Naturally, we support the full range of ELB features and options in the Cloudamatic toolkit’s “Basket of Kittens” stack descriptor language.
The Elastic Load Balancer service was launched by Amazon in 2009. While it has evolved incrementally over time, it does not support newer protocols such as HTTP/2, nor integrate with some AWS services such as Web Application Firewall and EC2 Container Service.
A Case to Study
In the fall of 2016, a Federal client for whom we manage a large AWS deployment was given an urgent requirement to blacklist a set of known hostile IP addresses and domains across their environment. While this is straightforward to support at the host level, there was a very specific requirement to stop all traffic from those sources at the network boundary. Most key applications were exposed to the public internet through ELB listeners, which have no filtering capabilities. Regular old Security Groups cannot be used to explicitly drop traffic from specific address. VPC Network ACLs can, however they are not intended for this purpose, and the small number of rules that can be defined makes this impractical.
Fortuitously, Amazon had just released Application Load Balancers. ALBs support many new features, such as path-based routing, HTTP/2, and, critically, integration with Web Application Firewall. WAF supports large-scale IP blacklisting, presenting us with a path forward for the client.
The existing application infrastructure was, of course, created and managed by Cloudamatic. We did not fancy replacing our ELBs with ALBs manually, leaving our deployment metadata out of sync and the new resources unmanaged. That meant we needed to teach our tools how to create and manage ALBs.
Once armed with the ability to build Application Load Balancers, we’d need to deploy into production infrastructure with minimal disruption. In some cases we can take the easy road: Build completely new stacks, identical except for ELBs in place of ALBs, and fall forward onto them by adjusting the appropriate DNS records.
That isn’t a great option for some of our longer-running applications, however. We have complex clusters of a particular Windows application that depends on locally-stored data, which are time-consuming to properly build from scratch and synchronize. It would be much nicer if we could insert our new ALB resource into the existing deployments, rather than incur a half day of downtime.
Once we have our hypothetical ALBs, setting up a static IP blacklist with WAF would take just a few minutes. But our requirements also included blocking certain domain names, and WAF, like many commercial firewalls, doesn’t pay attention to DNS. Our final step would be to engineer our own solution to cover the DNS angle.
In Part-II of this series we cover the more hard-core implementation details for ALBs and some creative ingenuity we employed to automate some customer specific requirements such as blacklisting of certain websites.