Ellipsis

The graphs I showed in part one are generated by performance output from a Perl script, check_access_log.pl (attached to this post). The performance data are then captured by pnp4nagios and saved into an RRD file. The default graphing templates provided with pnp4nagios v0.4 didn’t do justice to how interest these data are, so I wrote my own template (to be attached to a future post).

The usage for check_access_log.pl is:

Usage: check_log3.pl [ -v|--verbose ] -l|--logfile=file -m|--m=minutes [ -c|--critical=<threshold>(20) ] [ -w|--warning=<threshold>(10) ] [ -a|--activity=number_of_lines (100) ]

where:

• logfile is the path to your httpd access log
• minutes is how far back from the present you want to parse your log file (5 minutes by default)
• -c|-w represent the percentage of non-200 statuses are considered problematic
• --activity: only go to warning/critical if activity exceeds this number of lines per check. This prevent the check from going to warning/critical if you’ve taken it out of service.

The script requires:

• File::ReadBackwards; # allows the script to read backwards until the time period criterion is met
• Date::Manip;
• Nagios::Plugin;

Generally, the live httpd access logs will not be on your Nagios server so you’ll need to invoke this over NRPE.

When I started at $WORK we had little by way of a comprehensive monitoring system, and rectifying that quickly became my top priority. I rolled out Nagios, and needing a way to trend historical performance data, added "pnp4nagios"http://docs.pnp4nagios.org/pnp-0.4/start" on top of that. Over time I had a real desire to see what were the breakdown of 200s, 403s, 404s, 500s and other status codes being logged by our various web servers. What I wanted were graphs like this (oops, no traffic on the 20th),

 

or something that illustrated that it was okay when our backend Day Communique CQ5.3 publisher was running 80% error rates late in the afternoon (because our front-ends had cached all the 200 OK objects, but the 404s were still getting passed to the backend; in the morning, the publications flushed the front-end cache, and most 200 would go to the backend again):

 

Anyhow, setting up PNP4Nagios to generate these lovely, zoomable graphs will be the focus of my next post.

Last week I attended Surge up in Baltimore, and for the first time in over a decade I had the exhilarating sensation of being out of my depth. Since it was a small conference with some of the big players in web-scale applications and content delivery, there’s good reason for that, but it inspires me to pick up the pace in following some technologies I’ve only been tangentially aware of.

The most interesting talks were:

• Scalable Design Patterns, Theo Schlossnagle
• Enterprise solutions from commodity components, Brian Cantrill
• Growth at Gilt Group, Geir Magnusson, Jr.
• Why some architects almost never shard their applications, Baron Schwartz

I should find more to say about them, but the primary reason I haven’t been blogging much that last few months is that I try to take on too much writing too late in the evening, then run out of steam—so just take my recommendations for what they are.

All of the presentations are supposed to posted within a few weeks. I need to go back and watch a few that I missed. To wit, Artur Bergman on scaling Wikia, Justin Sheehy on Concurrency, Benjamin Black on Netflow (if time) and Neil Gunther on Quantifying Scalability.

At work we’re migrating several thousand research articles from Zope to another CMS. The CMS folks are taking care of moving the content, but when we’re done we’re going to institute a boatload of R=301 redirects from the old URLs to the new URLs.

RewriteMap is the accepted way with mod_rewrite of handling a lot of one-to-one mappings that don’t follow any particular pattern. What I figured out today was that I can use a rewrite map in a RewriteCond statement so I only do the redirect when there’s a match in the rewrite map lookup.

Here are some snippets from my httpd.conf to illustrate:

For testing, we’ll want logging:

RewriteLog /var/log/httpd/rewrite.log
RewriteLogLevel 2

Define the map we’re using. The content of the map is ‘old_uri new_uri’ with a space separating the two. Use .txt for testing, and the below we’ll convert to a DBM.

RewriteMap research_map txt:/etc/httpd/conf/research_map.txt 

Here, I got a hint from http://www.tunnell.org/blog_posts_view.php?blog_postid=3. Remember that the syntax for a RewriteCond is:

RewriteCond TestString ConditionPattern

TestString will be a map lookup of $1, where $1 is the match string of the following RewriteRule, expressed: ${research_map:$1}

For ConditionPattern, we will test if the TestString is lexically greater than ””, the empty string, which is what the map lookup returns when there’s no match. Expressed: >""

RewriteCond ${research_map:$1}  >""     # IE, if map result is greater than "" 

So if the URL starts with /research, then use the research_map value for the key $1 to redirect to new address

RewriteRule ^(/research/.*$) ${research_map:$1}      [R=301,L]

What we end up with is a few lines of configuration that quickly let me put in place 3342 new redirects. Here’s the whole stanza:

RewriteMap research_map txt:/etc/httpd/conf/research_map.txt 
RewriteCond ${research_map:$1}  >""     # IE, if map result is greater than "" 
RewriteRule ^(/research/.*$) ${research_map:$1}      [R=301,L]

# Same thing, but lookup with a trailing slash if there isn't one
RewriteCond ${research_map:$1/}  >"" 
RewriteRule ^(/research/.*[^/]$) ${research_map:$1/}      [R=301,L]

Lastly, converting the textfile to a dbm speeds up the lookup by at least an order of magnitude.