Apache Error “No space left on device: Couldn’t create accept lock”

We ran in to this issue earlier this morning. One of our web servers started hit heavily with spam in the form of automated web posting bots. Since we are hosting forums, 99.9% of the load was centered around MySQL. To quickly stop the spam without having to wait an hour for MySQL to shut down, we ran a “killall -9 httpd” to stop all new incoming web requests and existing request from processing.

I have done this numerous times in the past, more than I can possibly count without any issues. Now I realize that this is never the best way to stop a service, but surely it won’t harm a service as simple as Apache and is a quick emergency way of stopping the issue without having to reboot.

After the situation was under control, we tried restarting Apache

Even though it said it was starting successfully, it terminated immediately with the following message in /var/log/httpd/error_log

Now, most people would just reboot the server. Considering this has been running for 628 days and I did not have physical access to the server, I was very hesitant to do so. Clearly the problem was not related to disk space. Plenty of space there…

So I began searching… It turns out when I ran the killall, Apache left open several semaphores. I don’t have a sold grasp on exactly what they are, but based on some quick Googling, a semaphore is a way for two processes to communicate. Kindof like a socket. Apache uses them so parent and child processes can communicate. Using the ipcs -a command, you can see a list of all the semaphores that were causing the problem.

There was probably about 100 before I trimmed the output above (no sense in posting 10 pages of them). Anyway, long story short, the following command can be used in CentOS/RHEL to clear all of the apache-related semaphores.

Then rerun ipcs and you should see them all gone.

Once confirmed, you can try restarting Apache.

Success! Hopefully somebody else finds this useful and it helps save time.

Leave a Reply