We ran in to this issue earlier this morning. One of our web servers started hit heavily with spam in the form of automated web posting bots. Since we are hosting forums, 99.9% of the load was centered around MySQL. To quickly stop the spam without having to wait an hour for MySQL to shut down, we ran a “killall -9 httpd” to stop all new incoming web requests and existing request from processing.
I have done this numerous times in the past, more than I can possibly count without any issues. Now I realize that this is never the best way to stop a service, but surely it won’t harm a service as simple as Apache and is a quick emergency way of stopping the issue without having to reboot.
After the situation was under control, we tried restarting Apache
|
1 2 |
[root@localhost httpd]# service httpd start Starting httpd: [ OK ] |
Even though it said it was starting successfully, it terminated immediately with the following message in /var/log/httpd/error_log
|
1 2 |
[Tue Jan 17 10:40:47 2017] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run? [Tue Jan 17 10:40:47 2017] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.13808) (5) |
Now, most people would just reboot the server. Considering this has been running for 628 days and I did not have physical access to the server, I was very hesitant to do so. Clearly the problem was not related to disk space. Plenty of space there…
|
1 2 3 4 5 6 7 |
[root@localhost ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 427G 162G 244G 40% / tmpfs 16G 0 16G 0% /dev/shm /dev/sda1 190M 144M 37M 80% /boot /dev/mapper/vgraid-lvdata 493G 352G 116G 76% /d1 |
So I began searching… It turns out when I ran the killall, Apache left open several semaphores. I don’t have a sold grasp on exactly what they are, but based on some quick Googling, a semaphore is a way for two processes to communicate. Kindof like a socket. Apache uses them so parent and child processes can communicate. Using the ipcs -a command, you can see a list of all the semaphores that were causing the problem.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
[root@localhost ~]# ipcs -a ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x6c030690 98304 zabbix 600 527280 6 ------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 0 root 600 1 0x00000000 32769 root 600 1 0x00000000 4980738 apache 600 1 0x00000000 5013507 apache 600 1 0x7a030690 29261828 zabbix 600 13 0x530310b0 18219013 root 600 103 0x480310b0 18251782 root 600 9 [...] 0x00000000 38273132 apache 600 1 0x00000000 38305901 apache 600 1 0x00000000 38338670 apache 600 1 0x00000000 39387247 apache 600 1 0x00000000 39420016 apache 600 1 0x00000000 39452785 apache 600 1 0x00000000 39485554 apache 600 1 0x00000000 40566899 apache 600 1 0x00000000 40599668 apache 600 1 0x00000000 40632437 apache 600 1 0x00000000 40665206 apache 600 1 0x00000000 41779319 apache 600 1 0x00000000 42008696 apache 600 1 0x00000000 42041465 apache 600 1 0x00000000 42074234 apache 600 1 0x00000000 42991739 apache 600 1 0x00000000 43221116 apache 600 1 0x00000000 43253885 apache 600 1 0x00000000 43286654 apache 600 1 |
There was probably about 100 before I trimmed the output above (no sense in posting 10 pages of them). Anyway, long story short, the following command can be used in CentOS/RHEL to clear all of the apache-related semaphores.
|
1 2 |
for i in `ipcs -s | awk '/apache/ {print $2}'`; do (ipcrm -s $i); done ipcrm -m 0x63637069 |
Then rerun ipcs and you should see them all gone.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
[root@localhost httpd]# ipcs -a ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x6c030690 98304 zabbix 600 527280 6 ------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 0 root 600 1 0x00000000 32769 root 600 1 0x7a030690 29261828 zabbix 600 13 0x530310b0 18219013 root 600 103 0x480310b0 18251782 root 600 9 0x570310b0 18284551 root 600 1 0x00000000 20316176 root 600 1 0x00000000 20348945 root 600 1 0x00000000 20611090 root 600 1 0x00000000 20578323 root 600 1 0x00000000 20447252 root 600 1 |
Once confirmed, you can try restarting Apache.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[root@localhost httpd]# service httpd start Starting httpd: [ OK ] [root@localhost httpd]# ps aux | grep httpd root 14240 0.0 0.1 507400 39652 ? Ss 10:44 0:01 /usr/sbin/httpd apache 14722 0.7 0.1 515904 46208 ? S 10:50 0:08 /usr/sbin/httpd apache 14897 0.7 0.1 512364 41792 ? S 10:53 0:06 /usr/sbin/httpd apache 14999 0.5 0.1 517820 45860 ? S 10:57 0:03 /usr/sbin/httpd apache 15002 1.3 0.1 517380 46308 ? S 10:57 0:08 /usr/sbin/httpd apache 15020 0.6 0.1 518376 46340 ? S 10:58 0:03 /usr/sbin/httpd apache 15022 0.8 0.1 514460 54272 ? S 10:58 0:05 /usr/sbin/httpd apache 15030 0.8 0.2 518264 69844 ? S 10:58 0:04 /usr/sbin/httpd apache 15084 0.6 0.1 517512 46092 ? S 11:00 0:03 /usr/sbin/httpd apache 15097 0.5 0.3 588448 115032 ? S 11:00 0:02 /usr/sbin/httpd apache 15120 1.2 0.1 512132 40356 ? S 11:01 0:05 /usr/sbin/httpd apache 15124 1.2 0.3 547940 104588 ? S 11:01 0:05 /usr/sbin/httpd [...] |
Success! Hopefully somebody else finds this useful and it helps save time.