Ellipsis

This morning I was having trouble getting Apache (2.0.X) to restart. I was getting these error messages:

[Thu Apr 10 08:32:49 2008] [crit] (17)File exists: unable to create scoreboard "/var/private/logs/apache_runtime_status" (name-based shared memory failure)

and I kept deleting the offending file, and making sure that permissions all along the path were correct. No dice.

So I restarted Apache again running strace:

strace /usr/sbin/httpd2-prefork -X -Dprivate '-CPidFile /var/run/apache2.private.pid' -f /etc/apache2/httpd.conf

and saw output like this:

unlink("/var/private/logs/apache_runtime_status") = -1 ENOENT (No such file or directory)
open("/var/private/logs/apache_runtime_status", O_WRONLY|O_CREAT|O_EXCL, 0666) = 9
stat("/var/private/logs/apache_runtime_status", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
shmget(16908303, 40824, IPC_CREAT|IPC_EXCL|0600) = -1 EEXIST (File exists)
write(2, "[Thu Apr 10 08:18:46 2008] [crit"..., 168) = 168

Red Herring

Turns out that worrying about the filesystem was a distraction. The real issue is that the previous Apache had left behind shared memory segments that it couldn’t access anymore. Now I won’t pretend that I understand the ins and outs of shared memory on Linux, but a bit of Googling led me, fortunately to Sven Vermeulen’s blog, where he shared a similar experience on Solaris. I would’ve left nice comments on his blog, but as he doesn’t take comments, I’ll give a shout out to him here.

Anyhow, the ipcs and ipcrm commands come to the rescue:

# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x0102000f 99942402   root      600        40824      0

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x00001f58 0          root       600        0            0

Ah-ha—there’s shmid at 99942402. Let’s rm that:

# ipcrm -m 99942402
# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x00001f58 0          root       600        0            0

And apache started right up. Yea.

P.S. One find out more about a shared memory segment with something like: ipcs -m -i 99942402, which will report:

Shared memory Segment shmid=99942402
uid=0   gid=0   cuid=0  cgid=0
mode=0600       access_perms=0600
bytes=40824     lpid=6987       cpid=6987       nattch=0
att_time=Thu Apr 10 08:20:48 2008  
det_time=Thu Apr 10 08:21:55 2008  
change_time=Thu Apr 10 08:20:48 2008