Here at Trek10, I am frequently exposed to Kubernetes and EKS projects. To avoid costs and general AWS-specific administrative overhead, I frequently work with a Kubernetes cluster in my home lab. While Application Load Balancers (ALB) are the go-to when load balancing web application traffic within the AWS cloud, I need a more readily available solution when working on POCs in my home lab.
In this blog post, I’ll share the solution I came up with that uses an Nginx instance as a reverse proxy that doesn’t make use of any common Kubernetes ingress objects. In other words, I’ve created a quick way to load balance access to a Kubernetes service from a perspective outside of a cluster.
Full disclosure, two different types of popular Nginx ingress controllers already exist for Kubernetes that you can use to load balance requests across pods. These would be something you might use in a production environment if you chose not to opt for the AWS Load Balancer Controller Ingress.
You can find these two Nginx projects at the following links.
Despite their existence, I choose to use a different, easier way of sending requests to Kubernetes workloads in my lab environment. And just as a general preference, I like the idea of using an external load balancer over one provisioned in Kubernetes for the reason that doing so presents a system closer to what you might work with in a cloud environment (an AWS application load balancer).
With all that said, let’s jump into the nuts-and-bolts of what I’m babbling about.
I’ve created a script that runs an ephemeral Nginx Docker container that will proxy requests to services provisioned in Kubernetes. That’s it. Plain and simple but very useful when testing Kubernetes workloads.
Getting into the details, let’s kick this off by creating a Kubernetes deployment and an accompanying service. Apply the following manifest to your cluster.
Note that I make use of the spec.externalTrafficPolicy directive in my service manifest to ensure all traffic sent to a node stays “local” to that node. As in, a request routed to node1 (by the Nginx load balancer) won’t end up being routed to node2 (by kube-proxy).
Once applied, verify that your pods and services are provisioned and functioning properly.
$ kubectl apply -f simple.yaml
$ kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName
Note that my test cluster utilizes two worker nodes.
With our test infrastructure provisioned, let’s now look at how we would manually create an Nginx container to load balance connections to the NodePort services listening on TCP port 30080 on each worker node.
We first need to write a configuration file to disk for Nginx to utilize. Something like the following could be written to /tmp/conf/nginx.conf.
I’ve added some directives that instruct the requesting user-agent (browser) to disable caching such that a fresh response is received from the responding web server every time a request is made. Something that can help out quite a bit when testing minor code changes during development.
With our configuration file written to disk we are now ready to start an Nginx container (outside of a kubernetes cluster) that will mount and use this configuration. This can be accomplished via the following command.
Once running, this command will create a Docker container with an Nginx server that will listen on TCP port 30080 (on the host running the container) and will load balance requests made to this port and send them to TCP port 30080 on the cluster’s worker nodes (192.168.0.231 and 192.168.0.232).
Which ends up looking like the following.
Requesting the load-balanced URL several times will see us cycle through all of the pods in the deployment we provisioned.
Excellent! This is exactly what we’re looking for.
We could stop there and claim success but I figured I ought to accompany this blog post with a script that helps automate the process. Seems like the right thing to do. Someone might even find it useful!
You can use the following script to run an Nginx container in the same manner as described above with multiple listening ports and multiple Kubernetes back-end services.
You feed the script ports you want to listen on and connect with back-end Kubernetes services using the “-p” flag as well as your cluster’s worker nodes via the “-s” flag. You can use both flags multiple times.
Reference the following diagram for what this looks like once executed.
And that’s all folks! Thanks for hanging out with me for a bit. I hope you were able to take something from this post and find this technique useful.