Blog

Apache fails to start : No space left on device

Tags: accept lockApacheNo space leftsemaphore

Scenario:

Apache Semaphore issues

This is one of the rarest conditions which I came across. The webserver failed to load with following error messages in the server log file. The server had fairly good uptime and was a busy server.

[emerg] (28)No space left on device: Couldn't create accept lock
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[notice] Digest: generating secret for digest authentication ...
[notice] Digest: done
[warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[emerg] (28)No space left on device: Couldn't create accept lock

The error message
No space left on device: Couldn’t create accept lock

made me confused as there were no disk space or quota related issues.

 On a detailed analysis, it is found to be  an Apache semaphore issue.

 An  Apache semaphore is an inter-process communication tool that is used by Apache to communicate with its child processes. Semaphore marks memory locations as locked and may not release it upon completion of process. In most case, if parent process dies before the child.

Due to the marking of  too many memory locations as being used while they are not,  the system will run out of memory locations after some time. This can happen in very busy servers with high uptime. In the previous case, the error message indicates that the apache server fails to allocate semaphores for new child processes.

Check to see how many semaphores are currently in use. If Apache is running correctly, you should see something like this

root@server [/]# ipcs -s

------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 0 root 600 1
0x00000000 241893377 nobody 600 1
0x00000000 241926146 nobody 600 1
0x00000000 241958915 nobody 600 1
0x00000000 241991684 nobody 600 1

In an ideal condition, the Apache semaphores should be cleaned up once the webserver is stopped. If Apache is stopped, and you still see these semaphores, then it indicates that Apache has not cleaned up after itself, and some semaphores are stuck.  If the number of semaphores are high, it can lead to severe memory leakage and results resource non availability.

In my case, I got a large array of Semaphores even after the web server is down.

root@server [/]# ipcs -s

------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 0 root 600 1
0x00000000 242057217 nobody 600 1
0x00000000 242089986 nobody 600 1
0x00000000 242122755 nobody 600 1
0x00000000 242155524 nobody 600 1

Now the solution is safely remove the semaphores.

You can safely kill them by running this command for each semaphore id (in the second column)

ipcrm -s <semid>

Below is the command I used to delete the semaphore

root@server [/]# ipcrm -s 242155524

Once the semaphores are killed you need to check the queue size

root@server [/]# ipcs -s

------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 0 root 600 1

root@server [/]#

To destroy all semaphores, you can run this from the command line (with “nobody” being the apache-user:

for semid in `ipcs -s | grep nobody | cut -f2 -d" "`; do ipcrm -s $semid; done

or

for i in `ipcs -s | awk ‘/httpd/ {print $2}’`; do (ipcrm -s $i); done

Increase Apache semaphore limit

The permanent solution for any such issue would obviously be increasing the current limits. You can view the current parameters:

root@server [/]# ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 2048
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

To change these parameters, modify the file /etc/sysctl.conf and add the following lines:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

Then load these settings with the command:

sysctl -p

Category : Apache, Howtos, Troubleshooting