We ran in to this issue earlier this morning. One of our web servers started hit heavily with spam in the form of automated web posting bots. Since we are hosting forums, 99.9% of the load was centered around MySQL. To quickly stop the spam without having to wait an hour for MySQL to shut down, we ran a “killall -9 httpd” to stop all new incoming web requests and existing request from processing.
I recently replaced one of our cPanel DNS-only nameservers. Typically we run our DNS servers with 512MB of memory, because why would a simple DNS server need any more than that? I was shocked to see that cPanel, even the free DNS-only version, now requires 768MB of memory if you’re using CentOS7/RHEL6 or 1GB if you’re using CentOS7/RHEL7. If you are like us and are using a VPS to host your DNS, this can double your cost spent to maintain your DNS servers.
One of the websites we run serves a massive amount of static content. However, each page request needs to query a MySQL database to retrieve the storage location of the files. To reduce the load on our servers, we implemented an Nginx reverse proxy with caching. As our site grew, we quickly outgrew the system we were hosting the proxy on. At its peak time of day, this site is now serving over 600 requests per second for small image files averaging 15KB each. This is generating a massive amount of disk IO to the point where the disk was pegged at 100% utilized all the time, and starting to impact performance.
ClamAV is an open source anti-virus utility for Linux designed to detect viruses, malware, and our favorite – php script exploits. It does a great job at picking up php files containing backdoors, remote file managers, spam mailer bots, etc. We run this on all of our hosting servers daily. It is especially useful for detecting and removing files uploaded by the numerous exploitable Wordpress 3rd party plugins.
We recently began connecting the IPMI/remote access cards in our server farm. The remote access cards are a separate ethernet port that is always on, even when the server is powered off. You can connect in to this interface using your web browser to monitor the server’s status and issue minimal commands, such as power on and open a remote console. Because most of our servers are based on the Dell PowerEdge 1950 and PowerEdge 2950, they are running some slightly-older cards, the Dell DRAC 5.
As a follow up to our syslog sever documentation, we wanted to also document how to enable encryption on the syslog stream since private information, including credentials, could be getting passed from client to server in the logs. In this document, we will be using self-signed certificates, including a self-generated CA certificate.
The number of servers in our farm is continuing to grow. It’s becoming more and more difficult to monitor them all as closely as we would like. We decided that it’s time to set up as centralized location for log files to keep a closer eye on everything and allow us to easily develop our own reports and triggers against the logs. For this, we will be using rsyslog with a 3rd party program, LogAnalyzer. For the purposes of this document, we will assume that you already have a MySQL database configured running on a separate server.
As the battle to fight spam continues on the mail relay that serves our shared hosting services, we started working on a new way to stop spam from being sent. We have some customers who end up with compromised sites over and over again. These customers often don’t care that their site is infected get get irritated when we suspend their accounts. Some of them have asked us to simply remove email permissions from their accounts.
We have been fighting a lot of spam recently on our web hosting service. We decided the best route to go was to set up a mail gateway on a separate server and run spam scans on all outgoing mail with SpamAssassin to discard junk. This helps prevent our servers from appearing on blacklists and helps keep customers happy. This tutorial walks through the process we used to set up our mail gateway. We are running CentOS 6.6 x64 with postfix and SpamAssassin.
One of the most important aspects of system administration these days is monitoring the traffic on your server. Many hosting providers impose limits on how much you can transfer per month. If you go over these limits, it’s not really a problem from the data center’s point of view, but you will usually end up getting slapped with a fee per GB you transfer in excess of your limits. This can add up quickly if you have a busy site.