This morning I was having trouble getting Apache (2.0.X) to restart. I was getting these error messages:

[Thu Apr 10 08:32:49 2008] [crit] (17)File exists: unable to create scoreboard "/var/private/logs/apache_runtime_status" (name-based shared memory failure)

and I kept deleting the offending file, and making sure that permissions all along the path were correct. No dice.

So I restarted Apache again running strace:

strace /usr/sbin/httpd2-prefork -X -Dprivate '-CPidFile /var/run/apache2.private.pid' -f /etc/apache2/httpd.conf

and saw output like this:

unlink("/var/private/logs/apache_runtime_status") = -1 ENOENT (No such file or directory)
open("/var/private/logs/apache_runtime_status", O_WRONLY|O_CREAT|O_EXCL, 0666) = 9
stat("/var/private/logs/apache_runtime_status", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
shmget(16908303, 40824, IPC_CREAT|IPC_EXCL|0600) = -1 EEXIST (File exists)
write(2, "[Thu Apr 10 08:18:46 2008] [crit"..., 168) = 168

Red Herring

Turns out that worrying about the filesystem was a distraction. The real issue is that the previous Apache had left behind shared memory segments that it couldn’t access anymore. Now I won’t pretend that I understand the ins and outs of shared memory on Linux, but a bit of Googling led me, fortunately to Sven Vermeulen’s blog, where he shared a similar experience on Solaris. I would’ve left nice comments on his blog, but as he doesn’t take comments, I’ll give a shout out to him here.

Anyhow, the ipcs and ipcrm commands come to the rescue:

# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x0102000f 99942402   root      600        40824      0

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x00001f58 0          root       600        0            0

Ah-ha—there’s shmid at 99942402. Let’s rm that:

# ipcrm -m 99942402
# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x00001f58 0          root       600        0            0

And apache started right up. Yea.

P.S. One find out more about a shared memory segment with something like: ipcs -m -i 99942402, which will report:

Shared memory Segment shmid=99942402
uid=0   gid=0   cuid=0  cgid=0
mode=0600       access_perms=0600
bytes=40824     lpid=6987       cpid=6987       nattch=0
att_time=Thu Apr 10 08:20:48 2008  
det_time=Thu Apr 10 08:21:55 2008  
change_time=Thu Apr 10 08:20:48 2008  

As of today, I’ve restructure my work arrangment to 66% working from home for UCAR/NCAR , 33% working for EchoDitto. One thing I’ve found in the past is that my productivity can lag when I’m not in a traditional work environment, and working from home has led to weight gain and listlessness.

If I were insane, I’d do things the same way I had in the past, but I hope I’m not. So here’s how day one has gone:

• Got a good swim and bike in before work
• Set up a well lit office with lots of natural light and reasonable ergnomics (actualy, my chai sucks, my mouse is too low, and my second monitor is too old).
• Set up a sound system
• Warned my boys (at home with a nanny these days) of the dire consequences of intruding into my space
• Kept tabs of my work in real-time with the Harvest time-tracking app
• Wrote a short work week description to my bosses
• Kept my breaks short and focussed (morning coffee, lunch, afternoon snack)

  Still, I spent too much time following web trails, and not enough writing, writing, writing. So, since goals publicly stated are more likely to be realized than those that are not, I’ll use my blog for both some technical updates and productivity ones also. Now I’d write more but my nanny has to leave.

Today I wrote a longish paper in Textile before I recalled that I was going to have to post this to a Twiki. Dang. I can’t stand Twiki syntax. Anyhow, I was able to use the following “Find & Replace” commands in Textmate to convert Textile to Twiki syntax.

Links:

Headings

Lists

These replacements took care of 95% of the conversion, changing the blockquotes was easier done by hand than by trying to cook up a regexp for the two cases, as was putting in the <nop> by hand so OpenPGP wasn’t interpreted as a page link. Hurrah for Textmate!

This week I happened upon a client who was eager to solve a persistent problem with Microsoft’s Internet Explorer bombing when trying to POST content over HTTPS to a custom web application. The client sent me the thread from the trouble tracking system, and it was clear that they were already aware of the magic Apache mod_ssl incantation to address some of MSIE’s non-compliant behavior. Since they were already barking up that tree, I decided that they needed a better test bed to help confirm whether the problem was truly being addressed by whatever remedies they were hauling out. To put it another way, we couldn’t really apply any scientific method unless we could have a control case and an experimental case.

Since the problem was MSIE specific, I first needed a way to drive MSIE through some test cases and evaluate the results.

Getting a flawed version of MSIE

First, I had to get a sufficiently old version of MSIE, since late versions of IE6 and IE7 are all okay. To do this all safely and reproducibly, I’m running Windows under an instance of VmWare Server on a Linux host. To get things set up, I took care of the following:

• Install Windows 2000 Professional from CD (apply no security patches)
• Install Windows 2000 SP2 http://www.microsoft.com/windows2000/downloads/servicepacks/sp2/sp2en.mspx
• Install Microsoft Internet Explorer SP1 http://www.microsoft.com/windows/ie/ie6/downloads/critical/ie6sp1/default.mspx

(if you have WinXP CD, start there, since IE 6 first came out with Win XP)

That will give you MSIE 6.00.2800.1106, which failed miserably when I ran it through it’s paces to POST content over HTTPS.

Automating MSIE with Perl Win32::IEAutomation

Next, I needed to automate testing with MSIE, and to the rescue comes Prashant Shewale’s Perl module Win32::IEAutomation. To run the module, I did the following on my Win2k System

• Download and install ActiveState’s Active Perl 5.8.X: http://www.activestate.com/Products/Download/Download.plex?id=ActivePerl (Providing contact details are optional)

• Download and install Microsoft’s nmake.exe. See their Microsoft Knowledge Base, article 132084, and follow link to nmake15.exe. Then run the downloaded file, and move nmake.exe and nmake.err to c:\perl\bin.

• Run ‘cpan -i Win32::IEAutomation’ from the command line

At this point, I also installed CygWin and some decent editors to so some sane development and testing on the system, but that’s beyond the scope of this article.

Last, I wrote a variant on the following script to drive IE:

    use Win32::IEAutomation;

    # Set up variables
    $server_base="https://www.example.com";
    $wait=$ARGV[0];
    $now=localtime(time);
    $upload="C:\Documents and Settings\Peter Burkholder\My Documents\TextDoc.txt";
    $user="username\@email.com";
    $pass="password";
    $title="PeterB Test for $wait sec at $now";

    # Create new instance of Internet Explorer
    my $ie = Win32::IEAutomation->new( visible => 1, maximize => 1);

    # Goto Login page and Login
    $ie->gotoURL('https://example.com');

    $ie->getTextBox('name:', "username")->SetValue($user);

    $ie->getTextBox('name:', "password")->SetValue($pass);

    $ie->getButton('caption:', "Login")->Click;

    # Navigate to the add content page
    $ie->gotoURL('https://example.com/home/content.php');

    $ie->getButton('caption:', "Create new")->Click;

    ## Fill in Content Page
    $ie->getTextBox('name:', "name")->SetValue($title);
    $ie->getSelectList('name:', "company_id")->SelectItem("ACA");

    # IE fails on 6.00.2800.1106 whether or not a file is uploaded
    # Replace the
    # $ie->getTextBox('name:', "filename")->SetValue($upload);
    $ie->getTextArea('name:', "note")->SetValue("Sample comment on the upload");

    # Now we sleep to see at least 30 seconds to get the Post error, then click the "Save" button
    sleep($wait);
    $ie->getButton('caption:', "Save")->Click;

    # Summarize the output and quit IE so we always start from a known state
    $output=substr($ie->PageText(),0,40);
    print $output;

    $ie->closeIE();

The code starts up IE and walks it through the first few panes of the application until the point where the error has been known to occur. It’s evoked as, say: perl ieautomate.pl 5 where the last argument is the number of seconds to wait before the ultimate submit. When run with a short wait, like 5 seconds, the content is successfully posted. With a wait of 30 or 40 seconds, the submit fails.

Running this is really cool, like some poltergeist has taken over the machine. I can’t wait to use Win32::IEAutomation to check airline ticket prices, etc.

Diagnosing the SSL problems

This breaks down into two steps, a) getting VmWare host-only networking set up to route through the host so we can then b) run sslsniff on the traffic and look inside the packets.

A) Getting VmWare routing set up

Thanks to the folks at Cyberciti.biz for getting me on the right track. Their post on the matter is largely correct except that:

• You need to run: echo 1 > /proc/sys/net/ipv4/ip_forward and add that config to /etc/sysctl.conf

    option routers 192.128.2.1;
    option nameserver (_real ip of nameserver_)

It turns out this step is totally unnecessary. I’d intended to use the routing from the host’s eth0 interface to the guest’s vmnet1 subnet to run Mike Benham’s sslsniff. While SSLSniff works great in such a setup, if sufficently munges up the SSL traffic that it doesn’t aid in addressing the MSIE problem, in fact, it pretty well makes it go away.

B) Analyzing traffic with Ethereal/Wireshark

Ack, I’m getting tired so sorry this last part is so lame. What it comes down to is that three test cases were sufficient to reveal the crux of the problem.

1. Firefox SSL POST—when using Firefox and taking about 30 seconds to fill out the form that gets POSTed, one can see ‘Encrypted Alerts’ coming down from the server about every ten seconds. The alerts are probably change_cipher_spec or more likely a close_notify. When the POST is sent, Firefox starts with an SSL ClientHello and sets up a whole new SSL session

1. Automated MSIE post with no delays—when POSTing the form from the robot with no waits, everything works just great.

1. Automated MSIE post with a 30s delay—while the robot is waiting to POST the form, the same Encrypted Alerts come down from the server, and the client responds with ACKs. But when the form is POSTed the client is trying to re-use the same SSL connection. The server simply replys with ACKS, and MSIE barfs

In case number 3 it’s pretty clear that the Magic Apache mod_SSL Incantation is not working, as close_notify messages are still getting sent from the server.

Over and out.

Today I interviewed for a position with University of Maryland Office of Information Technology and the University’s College of Chemical and Life Sciences. As part of the interview process I was able to do a short presentation, and I chose to provide an overview of the web content management systems (wikis, blogs, cms’s and backpack) that I’ve worked with, and the role I think they can play in a univeristy environment beyond the strictly instructional.

Some of the resources I referenced in the talk are:

• Wikipedia (of course)
• the wikis hosted at swiki.dlese.org
• RealClimate.org, an excellent blog on climate science
• Buzz, the science blog and the Science Museum of Minnesota
• BZST, a blog by UMd Professor Galit Shmueli
• Plone, the Zope-based CMS
• the Plone sites hosted at www.dlese.org
• the Joomla sites at the College of Chemistry and Life Sciences, the Graduate Students’ Organization and the Doyle Research Group
• 37Signals, the web app development group responsible for Ruby on Rails and Getting Real
• Backpack the place to put all your stuff, also from “37Signals”

Lastly, one can download the PDF version of my presentation

I’ve migrated a lot of my old sysadmin stuff from my old host. This includes the following stuff

Presentations

• UCAR Web Advisory Group Shibboleth Presentation, October 2005
• UCAR Content Management System Forum, October 2004
• Web Application Security. A presentation for the Denver chapter of ISSA, 10 April 2002.
• SSH & SSL for SysAdmins. A presentation for University of Washington NetSys/LanAdmin, 24 January 2002.
• SSH & SSL. A presentation for the Colorado chapter of SAGE (Co-SAGE)”, 14 Nov 2001.

Notes

Some notes on work I do, which may or may not be useful to others

• How to run your own CA for launching SSL attacks
• PortSentry
• VPNs using Free S/WAN

Articles

• SSL Man-in-the-Middle Attacks 1 Feb 2002. An evaluation of several SSL attacks, includes a modification to Dug Song’s webmitm exploit.

RSS Scraper Job Feeds

Since I’m looking for work with specific institutions of interest in the Washington, DC, area, I thought I could track jobs more easily if I could subscribe to the relevant RSS feeds. Alas, although any institution now has its “Job Opportunities” web page, they seldom have RSS feeds of those pages. So I decided to work out my Ruby and Regular Expression skills and build my own feeds. These are built from RubyForge RSSscraper, written by Monster or <a href=”http://www.dice.com”>Dice or their ilk, so I can’t compare this to what they offer. But I find it useful so far, although scraping HTML from the NSF has been really challenging. I’d recommend that one sign up with their email job notification service.

RSS Job Feeds

• AAAS Jobs
• NSF Jobs
• Univ. of Maryland, College Park, Jobs

A few months back I was working heavily in Plone—so much so that I did a lengthy presentation to fellow sysadmins and web developers at UCAR

At this point I’ve grown weary of Plone. A CMS is not generally needed in most environments; a Wiki serves just fine. Further, the ZODB and the Zope stack made me long for just having everything on the filesystem and Subversion. One get that to work for Zope, but it takes some jumping through hoops.

Further, I didn’t care much for the Zope/Plone blog offerings: simpleBlog and Quills are what I tried.

So, all the article posts from April and May, 2006, are Plone ports—the formatting looks okay (I pasted in as straight html), but I may have missed things.

On the hosting end, I’ve still not switched. Thanks to Antoni Cangiano I think I’ll be trying Site 5 next.

• M4 and procmail testing
• M4 and httpd.conf maintenance
• M4 and a complex build system for multiple environments

I think m4 has a lot to offer system administrators and web developers who need to maintain configuration files in a manner that are beyond the scope of simple sed, awk or cfenging scripts.

More to come on this topic

My life is exciting enough that I could spend Friday evening working on my resurrected site.

I’d intended to document the experience well enough that I could update the various wikis on Typo+Dreamhost, but it got so late that my brain was too fried to do anything but randomly edit previous commands in the hope that something would work. I love to tinker around at night, but my brain stalls out a lot earlier than it used to.

The main sticking points I ran into were:

• The SVN checkout of the Rails1.1 typ version took about two bowls of cereal to complete
• The rake migration calls do nothing—I never timed them but after about 5 minutes I start hitting Ctlr-C. I didn’t dig into the strace of rake too deeply, but just set up the databases from within the mysql client with ‘source db/schema.mysql.sql’
• I was getting mysql server not found errors—because I had a TAB character after my hostname in my database.yml file. Why are computers so damn literal?
• I was getting 500 server errors, so I applied the RailsFCGIHandler tweak from Alex Young’s Blog

Of these, only the rake problems merit documenting on a Wiki—and I’ve done so at Typo Trac Wiki

This was pointed out to me not long ago. With the recent versions of the Firefox Web Developer Extension you can get on-the-fly breakouts of CSS. Just hit Ctrl-Shift-F to see which CSS to tweak.

For the Plone site I’ve been developting for the WAG site, hovering over the UCAR tab shows that I could tweak the tab display by using the CSS element #portaltab-ucar