We ran in to this issue earlier this morning. One of our web servers started hit heavily with spam in the form of automated web posting bots. Since we are hosting forums, 99.9% of the load was centered around MySQL. To quickly stop the spam without having to wait an hour for MySQL to shut down, we ran a “killall -9 httpd” to stop all new incoming web requests and existing request from processing.
I recently replaced one of our cPanel DNS-only nameservers. Typically we run our DNS servers with 512MB of memory, because why would a simple DNS server need any more than that? I was shocked to see that cPanel, even the free DNS-only version, now requires 768MB of memory if you’re using CentOS7/RHEL6 or 1GB if you’re using CentOS7/RHEL7. If you are like us and are using a VPS to host your DNS, this can double your cost spent to maintain your DNS servers.
One of the websites we run serves a massive amount of static content. However, each page request needs to query a MySQL database to retrieve the storage location of the files. To reduce the load on our servers, we implemented an Nginx reverse proxy with caching. As our site grew, we quickly outgrew the system we were hosting the proxy on. At its peak time of day, this site is now serving over 600 requests per second for small image files averaging 15KB each. This is generating a massive amount of disk IO to the point where the disk was pegged at 100% utilized all the time, and starting to impact performance.
ClamAV is an open source anti-virus utility for Linux designed to detect viruses, malware, and our favorite – php script exploits. It does a great job at picking up php files containing backdoors, remote file managers, spam mailer bots, etc. We run this on all of our hosting servers daily. It is especially useful for detecting and removing files uploaded by the numerous exploitable Wordpress 3rd party plugins.
We recently began connecting the IPMI/remote access cards in our server farm. The remote access cards are a separate ethernet port that is always on, even when the server is powered off. You can connect in to this interface using your web browser to monitor the server’s status and issue minimal commands, such as power on and open a remote console. Because most of our servers are based on the Dell PowerEdge 1950 and PowerEdge 2950, they are running some slightly-older cards, the Dell DRAC 5.
As a follow up to our syslog sever documentation, we wanted to also document how to enable encryption on the syslog stream since private information, including credentials, could be getting passed from client to server in the logs. In this document, we will be using self-signed certificates, including a self-generated CA certificate.