Apache Error “No space left on device: Couldn’t create accept lock”

We ran in to this issue earlier this morning. One of our web servers started hit heavily with spam in the form of automated web posting bots. Since we are hosting forums, 99.9% of the load was centered around MySQL. To quickly stop the spam without having to wait an hour for MySQL to shut down, we ran a “killall -9 httpd” to stop all new incoming web requests and existing request from processing.

I have done this numerous times in the past, more than I can possibly count without any issues. Now I realize that this is never the best way to stop a service, but surely it won’t harm a service as simple as Apache and is a quick emergency way of stopping the issue without having to reboot.

After the situation was under control, we tried restarting Apache

[root@localhost httpd]# service httpd start
Starting httpd:                                            [  OK  ]

Even though it said it was starting successfully, it terminated immediately with the following message in /var/log/httpd/error_log

[Tue Jan 17 10:40:47 2017] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Tue Jan 17 10:40:47 2017] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.13808) (5)

Now, most people would just reboot the server. Considering this has been running for 628 days and I did not have physical access to the server, I was very hesitant to do so. Clearly the problem was not related to disk space. Plenty of space there…

[root@localhost ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             427G  162G  244G  40% /
tmpfs                  16G     0   16G   0% /dev/shm
/dev/sda1             190M  144M   37M  80% /boot
/dev/mapper/vgraid-lvdata
                      493G  352G  116G  76% /d1

So I began searching… It turns out when I ran the killall, Apache left open several semaphores. I don’t have a sold grasp on exactly what they are, but based on some quick Googling, a semaphore is a way for two processes to communicate. Kindof like a socket. Apache uses them so parent and child processes can communicate. Using the ipcs -a command, you can see a list of all the semaphores that were causing the problem.

[root@localhost ~]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x6c030690 98304      zabbix     600        527280     6

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 32769      root       600        1
0x00000000 4980738    apache     600        1
0x00000000 5013507    apache     600        1
0x7a030690 29261828   zabbix     600        13
0x530310b0 18219013   root       600        103
0x480310b0 18251782   root       600        9
[...]
0x00000000 38273132   apache     600        1
0x00000000 38305901   apache     600        1
0x00000000 38338670   apache     600        1
0x00000000 39387247   apache     600        1
0x00000000 39420016   apache     600        1
0x00000000 39452785   apache     600        1
0x00000000 39485554   apache     600        1
0x00000000 40566899   apache     600        1
0x00000000 40599668   apache     600        1
0x00000000 40632437   apache     600        1
0x00000000 40665206   apache     600        1
0x00000000 41779319   apache     600        1
0x00000000 42008696   apache     600        1
0x00000000 42041465   apache     600        1
0x00000000 42074234   apache     600        1
0x00000000 42991739   apache     600        1
0x00000000 43221116   apache     600        1
0x00000000 43253885   apache     600        1
0x00000000 43286654   apache     600        1

There was probably about 100 before I trimmed the output above (no sense in posting 10 pages of them). Anyway, long story short, the following command can be used in CentOS/RHEL to clear all of the apache-related semaphores.

for i in `ipcs -s | awk '/apache/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069

Then rerun ipcs and you should see them all gone.

[root@localhost httpd]# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x6c030690 98304      zabbix     600        527280     6

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 32769      root       600        1
0x7a030690 29261828   zabbix     600        13
0x530310b0 18219013   root       600        103
0x480310b0 18251782   root       600        9
0x570310b0 18284551   root       600        1
0x00000000 20316176   root       600        1
0x00000000 20348945   root       600        1
0x00000000 20611090   root       600        1
0x00000000 20578323   root       600        1
0x00000000 20447252   root       600        1

Once confirmed, you can try restarting Apache.

[root@localhost httpd]# service httpd start
Starting httpd:                                            [  OK  ]
[root@localhost httpd]# ps aux | grep httpd
root     14240  0.0  0.1 507400 39652 ?        Ss   10:44   0:01 /usr/sbin/httpd
apache   14722  0.7  0.1 515904 46208 ?        S    10:50   0:08 /usr/sbin/httpd
apache   14897  0.7  0.1 512364 41792 ?        S    10:53   0:06 /usr/sbin/httpd
apache   14999  0.5  0.1 517820 45860 ?        S    10:57   0:03 /usr/sbin/httpd
apache   15002  1.3  0.1 517380 46308 ?        S    10:57   0:08 /usr/sbin/httpd
apache   15020  0.6  0.1 518376 46340 ?        S    10:58   0:03 /usr/sbin/httpd
apache   15022  0.8  0.1 514460 54272 ?        S    10:58   0:05 /usr/sbin/httpd
apache   15030  0.8  0.2 518264 69844 ?        S    10:58   0:04 /usr/sbin/httpd
apache   15084  0.6  0.1 517512 46092 ?        S    11:00   0:03 /usr/sbin/httpd
apache   15097  0.5  0.3 588448 115032 ?       S    11:00   0:02 /usr/sbin/httpd
apache   15120  1.2  0.1 512132 40356 ?        S    11:01   0:05 /usr/sbin/httpd
apache   15124  1.2  0.3 547940 104588 ?       S    11:01   0:05 /usr/sbin/httpd
[...]

Success! Hopefully somebody else finds this useful and it helps save time.

Leave a Reply