I am proud to say that this week I experienced my first real home production issue. It was Tuesday night; I had just gotten home at around 1900 hours after dealing with some issues at work and having an after-hours teleconference with some Australian representatives.
After I hastily made some food in the kitchen, I sat down for what I hoped would be a quiet evening of watching the X Files. Unfortunately my hopes were soon destroyed by a unusually frantically blinking Ethernet hub on my desk. At first I wasn't sure what was causing the spike in network activity, so I thought, "Maybe if I wait, it will subside." I was wrong.
Given I am running a number of servers, I decided to take a look at some of my various daemons to see if any of them are the culprit. My first stop was httpd2, which is a web server. Taking a look at its status, I could see that it was dealing with 50-80 requests concurrently when it would usually only handle at most five at a time with my usual traffic.
That's odd, I thought. I took a look at the logs to see what everyone was so frantically requesting. I discovered that my HTTP server was being bombarded with requests outside of my domain.
For those of you who don't know what it is that HTTP servers do, here is a rough explanation. HTTP traffic is plain text with addresses. You know these addresses well; you use them every time you type one into your browser's address bar. When you type in an address and press enter, the series of tubes that make up the Internet send your request to the appropriate server (exactly how is beyond the scope of this topic). The server replies with a web page which your browser displays. The server should only reply to web page requests within its own domain.
This brings us to the concept of proxy servers. A proxy is a special kind of HTTP server that receives requests from any site on the Internet and, if it doesn't have the web page stored locally, it will go and get it. So if you had a proxy and you wanted to see welbog.homeip.net, then your browser would ask your proxy for welbog.homeip.net. Your proxy would see that this isn't a local file, so it would go ask the real welbog.homeip.net for the page, and then forward that back to you. Proxies let you indirectly get anything from the Internet — very, very useful if you want to mask your IP address and any other identifying information.
I'm fairly certain you can see where I'm going with this. As you have probably guessed, my HTTP server was set up as a proxy: one that anyone could use. Somehow its address had found its way into a network of open proxies and was being abused by users of that network. All of this network traffic was caused by people trying to mask their identities online by using my server to browse the Internet.
The funny thing is there was nothing explicit in my server's configuration that made it a proxy. It defaulted to being a proxy: I had to add an instruction to tell it to not be a proxy. It makes me wonder what kind of programmer would have set it up that way in the first place.
My server is safe now with the resolution of my first production issue. It's not too much to hope that it is my last, is it?