Ellipsis

More from OSCON 2008

Peter Burkholder — Tue, 12 Aug 2008 05:32:00 -0700

WEDNESDAY

Full stack introspection

Theo Schlosssnagle (OmniTI Laurel Md), jesus at omniIT

Author of “Scalable Internet Architectures”

Whole stack introspection

MacOSX, BSD, Solaris
Paul Fox working on Linux port. Not in kernel, so CDDL/GPL, tension not a worry

DTrace approaches

Start w/ User
Sorcery (not recommended) Twiddle to fix, then verify. Starts w/ Hypotheses
Experienced. Starts in middle

It’s very important to know your optimum operation before you can diagnose anomalous behavior.

SysTap on Linux is way, way behind. Requires righting code and sticking in as module—not a way to work on production boxes.

MacOsX is very well instrumented.

Then follows a masterful demonstration of real-time system diagnoses. Obviously this is not something one learns in an afternoon, but certainly is more fruitful (and probably no harder) than what I hack up in grep, perl, or what not to figure our what’s going on in VKS, or ps, or logfiles.

Take aways:

Dtrace for Linux may be in the offing. Dtrace awesome. Watch for it.
I must better understand our architecture from ACE to backends.

Groovy vs. JRuby

A very good, well-organized talk. Both environments are ready to use. Groovy goes to Jars. Performance used to be 20% – 90% of pure Java, but rapidly closing. JRuby may make Java integration in the common cases but Groovy handles the edge cases better. Groovy language still a little immature, e.g. Regexs are not transitive. Groovy allows typing (which can be handy), but JRuby does clean duck-typing. JRuby has all Ruby libraries. Rails works in J2EE stack.

CouchDB.

Freakin’ awesome. Written in Erlang (BTW, Ericksson’s switches running erlang advertize 9 9s (0.3 s annually). I’ll need to write more about it later.

OpenSource Virtualization:

Completely lame presentation. One needs to learn that cutesy slides will emphasize, not obscure, your inexperience.

VmWare ESXi will be free with a non-deprecated API, announced June 22.

VirtualBox

Maps to VmWare Workstation, Player, Fusion, Server
Tools -> guest additions
vmx -> xml
has Seamless (Unity) latyer
harder: interfaces, network, disk, audio, USB.
very similar across OS’s, Solaris Host
Can run headless and over RDP.

Xen. Etc., but I was getting too annoyed by the talk so I caught the rest of the lightning talks and Google open source update.

Google Open Source Update

Brian DiBona and TKTK

Melange
Google Highly open participation contest ==
GHOP
DROP

Bofs and Parties

System Administration best practices BOF. Nothing new learned, but maybe I made some contributions.

Met a friendly pair of brothers from Baltimore. Mark and Curt Tilmes. Mark works at UMD (five weeks of vacation a year sounds pretty sweet). We went down to the OpenSolaris party at the DoubleTree, which was too redolent of an overamped frat party. The MindTouch people are having a great party up on the top floor the Red Lion, with a tight live band, nice atmosphere and a sweeping view of Portland across the Willamette. Too bad I’m too tired to stay here long and that it’s not better attended.

THURSDAY

New York Times intro to AWS

Brian Gottfried

Really needed PDFs from 1851-1981
Dynamically generate or… pregenerate 11 million PDFs and store statically.
Copy source to S3 store PDFs in S3 (which has been reliable except for last week)
4.3Tb of 20M files
Parallelization. Into pdfs, so map not reduce.
Non Hadoop part: read metadata, load TIFsS, scale arrange, generate PDF, write PDF
Minimal use of HDFS
Took previous code and conv. to S3 thanks to JetS3S

MapReduce

Map takes a Key/Value pair and output Key/Value pair
Reduce takes key & list of assoc. values and combines

HDFS (Yahoo! inc)

Cool, scary. Supports mapreduce. Need to read up.

Pushmi for Subversion

Subversion replication. Probably not relevant to me with Git coming onto our scene.

Configuration management

Commerical products are archaic and monolithic *Tivoli, CA Unicenter

Help desk, ticketing, asset management, a lot of things badly.

Puppet bcfg2, pikt, cfengine,

Measuring success.

Availability
Reduction of incidnets
Rate of failed change
Build time and speed to market
Compliance to SOE, policy or standard

NewStuff:

Puppetshow A Gui for history and nodes
iClassify (OS)
Puppetview Logger (simple PHP app)

Subnode (VMs and containers)

Zones look like resources
insides zone looks like a node (but behave differently

Other stuff: (see BOF below)

webrick – not for more 25 hots. 6000 hosts w/ Mongrel. Scale like any https. w/ loadbalancer.
puppet query against DB or LDAP.

Supervisor

Python. Built on medusa. Comparable: deamontools, launchd, runit but built for control of customer processes not pid 0.

supervisord forks arbitrary programs and child (including apache—which you run in FG since it BGs itself).
supervisorctl on same machine or over xmlrpc.
ini config file
captures child stdin/stdout and logs and rotates
users cannot stop/start arbitrary processes, only predefined processes
open or by authentication.
xmlrpc for your own supervisorctl.
single-process, thread. Your processes cannot block.
event-notification system
- in any language
- can spawn fastcgi processes
Can supervisor replace our bad sudo moderated scripts ??

Puppet BOF

TKTK from U. Georgia Board of Regents
TKTK from EngineYard
Luke Kanies
James Turnbull from Australian bank (today’s presenter)
TKTK from ????
Me

Puppet is in Git now. Puppet purge nop reports on all aspects of system that aren’t managed. Ralsh takes purge report and generates manifests. Scale by having one puppetmasterd dedicated to CA work and put the others behind LB as fileservers and manifest delivery. Stanford’s configuration takes 30s to parse about 1m to implement.

Svn pre-hook scripts can run parse check. Git lets you run pre-hook on local devel repository and again on core repository

Puppet under pressure to use XMPP instead of HTTP. New release reduces time by 70% by using YAML instead of Marshall.

Automateit Lightweight config management?

RALSH resource abstraction layer shell

FRIDAY

Keynotes

Mostly ho-hum. Microsoft’s opensource VP walked into the lion’s den and emerged mostly unscathed.

Sun’s TKTK on today’s languages. Reference to langpop.com language ranking based on. O’reilly book sales heat map. Tiobe index.

"I'd rather drive nails into my head" than code in PHP. (I didn't know they had over 5000 entries in their global namespace). Languages to watch: Groovy, OCaml (fast!), Erlang (stable, concurrent),  Scala (see Twitter), Fan.

Terms to look up: “Monkey Patching”

Java isn’t going away because the JVM and APIs are so mature, so the environment is being extended to Ruby, Python, Groovy, etc.

NASA

Curt Tilmes at Goddard

Data: Growth 3.2 tB/day; Archive 4.9PB; daily distribution processing 4.2 Tb

MODIS 250m resolution Level 1, raw Level 2, gridded Level 3, best 8 day w/o clouds if possible

...

http:/www.opensource.org/licenses/nasa1.3.php http://opensource.sfc.nasa.gov (Goddard specific, but other centers similar)

Curt gave a nice overview of opensource utilization at NASA and their attempts to contribute back. I would have liked more specifics on his struggle just to release a CPAN module. No specific technology takeaways, but I would like to see NIH/NCBI make a more determined effort to contribute on our periphery activities.

RT3/4

I tackled Jesse Vincent. between sessions. Told him about my RT to JIRA project and my reluctance about it.

RT4 is coming out but is going to be a deep rewrite with Jifty, new DB schema, AJAXy and drag-drop workflow scripting generation. No release date set.

rt3.8 has an ‘extract header’ extension for linking into other ticket systems, which could be useful to me.

REST apis have been growing and are supported. Not XML based, can be used from command line.

Base level support is $5k. Gold is $35k. But is based on ticket numbers.

Open Source Electronics

Tom talked about using open source tools to develop your own open-source hardward projects. Recommended references:

- Art of Electronics - OpAmps for Everyone - MIT OpenCourseware

OpenSource CAD. Start with Paper. Software: gEDA/gaf + PCB (first one I picked). Should be easy.

gschem gsch2pcb to circuit baord

PDB printing is clean fast cheap, credit card. www.4pcb.com www.sunstone.com, Alberta printed fast and cheap only two layer

Circuit simulation: SPICE. closed source freeware: LTSpice. NGspice part of gEDA project.

Digital simulation: Verilog.

Failures

Therac massive radiation overdose. Removed hardware interlocks. Killed six people. ETc. with space program, electric utilities, and so on.

Main points:

test
test before launch
test end to end
listen to your sysadmins

Buzzwords

open authentication
xmpp
jbod
json
erlang

Macports without root on Leopard (and Tiger)

Peter Burkholder — Mon, 11 Aug 2008 11:02:00 -0700

I don’t have root on my desktop Mac at $WORK, even though I have root on about 1200 production systems. sigh

So, I asked my Mac administrator to install the XCode development tools. Once that was done, all I needed to was the following.

Download the MacPort.tar.bz source distribution: http://svn.macports.org/repository/macports/downloads/MacPorts-1.6.0/MacPorts-1.6.0.tar.bz2

Note: For Leopard, one needs to apply the patch from http://trac.macports.org/changeset/32105

Run the following:


  tar -jxf MacPorts-1.6.0.tar.bz2 && cd MacPorts-1.6.0

  PATH=/usr/bin:/usr/sbin:/bin:/sbin ./configure \
  --prefix=$HOME/macports \
  --with-install-user=`id -un` \
  --with-install-group=`id -gn` \
  --x-includes=/usr/X11R6/include \
  --x-libraries=/usr/X11R6/lib \
  --with-tclpackage=$HOME/macports/share/macports/Tcl

  make
  make install

Add something like the following to your Mac’s .bashrc>


  export PATH=$HOME/macports/bin:$HOME/macports/sbin
  export MANPATH=$HOME/macports/share/man:$MANPATH
  export PERL5LIB=$HOME/macports/lib/perl5/5.8.8:$HOME/macports/lib/perl5/vendor_perl/5.8.8:$PERL5LIB

Run a selfupdate: port selfupdate
Now you do can do cool things like learn Scheme over lunch with: port search scheme; port install mzscheme

I’ve added this to the MacPorts wiki

Oscon 2008 Day One report

Peter Burkholder — Mon, 21 Jul 2008 22:03:00 -0700

Outbound: Spotted at least two OSCON attendees at Dulles Airport. One carrying a dynamic OLED keyboard (Optimus Maximus) that can display different character set imaginable (well, 113 different ones at least), another hard at work on her OLPC XO. I’m sure there were others, but better camouflaged. The flight was late, and cramped, and I was keenly aware of United’s 33-inch seat pitch versus the 33 inches one gets on Frontier Airlines.

The keyboard bearer was one convivial Jacob coming in from http://www.thinkgeek.com, and we together figured out the Max light rail in towards the convention center.

Monday: At breakfast I got an inkling that OSCON attendees are more sociable than their USENIX/SAGE counterparts. At least it wasn’t pulling teeth to engage in a conversation. Joel Noble of Caring Family described his group’s project to bring social networking to the elderly via simple pen & paper interfaces. Fred Meyer of USA Today was from my neck of the woods, and a fellow bike commuter, and we ended up walking Portland in the evening. As for the sessions….

Mastering Perl: brian d foy got down to some essentials of his book of the same name. Takeaways: Profiling can be useful, Benchmarking is less useful than one thinks (unless done correctly). Configuration is good. As is logging and persistence.

I then succumbed to my passions instead of my rational side—and jumped into the Arduino tutorial “Making Things Blink”. What a trip! Michael Dory, Adam Simon, and Scott Varland took us on a whirlwind introduction of the remarkably intuitive Arduino microcontroller system. Their kit included the Arduino Diecimila board, a interface shield, microbreadboard, LEDs, potentiometers, wiring, and force transducers. Oh, and a copy of “Making Things Talk” by Tom Igoe. Soon I was tweaking their circuit and code examples to implement my own riffs on their tutorial project, and by the end I had completed their digital ‘Etch-a-Sketch’ project

I’m excited to do more—and not for any practical reason. It’s just somehow exciting to get back to hardware, real transistors, resistors and diodes, after a generation of computing advances have made hardware less and less accessible at any level beyond swapping out whole components.

Peter out.

This American Life and Amazon Web Services

Peter Burkholder — Sat, 07 Jun 2008 10:03:00 -0700

Dear Ira Glass and Elizabeth Meister,

Today I was listening to your “This American Life” podcast and the prelude included a fund campaign to support your podcasts. Being a dutiful listener, and more than that, a fan committed enough to attend a live TAL show, I’m going to go beyond making a $5 donation, and help you save $70,000 per year.

Today you said that on average you support 400,000 download of your podcast per week (or 1.73 million a month) at a bandwith cost of $152,000 per year (or $12670 per month). That’s a situation not unlike the one we faced at EchoDitto when Rosie O’Donnell, or most prominent client, had b/w charges going through the roof when her podcasts were in the news last year. Our solution then was to move using media hosting through Amazon’s Web Services’ (AWS) Simple Storage Service (S3). Before we get into implementation, let’s look at cost for using S3:

STORAGE and UPLOAD:

Typical podcast size: 28M
Podcast storage for one year’s worth of podcasts (approx. 40 new podcasts per year): 1.1Gb
Monthly cost for 1 year’s podcast storage with S3 (1.1.Gb * $0.15 per GB per month): 16¬¢
Monthly upload costs (10¬¢ per Gb): 1¬¢
Subtotal storage and upload per month: 11¬¢
or, effectively nothing.

DOWNLOAD:

Bandwidth: 400,000 downloads per week of 28Mb podcasts: 10937 Gb/week, or ~50 Tb/Month
Requests: 400,000 * 4.33 : 1,732,000 GETs per month

And here I fall back on the AWS monthly calculator to come up $6912 per month, or $83,000 per year, an annual savings or $69,000

Implementation

Sign up for AWS with your WBEZ credit card
Sign up for S3, and get your AWS_ACCESS_KEY_ID and your AWS_SECRET_ACCESS_KEY
Download and install S3Fox for Firefox, and configure S3Fox with the keys from the step above
Use S3Fox to create a bucket called, say, media.thisamericanlife.org
Use S3fox to upload a podcast with an object key, say, ‘podcast/332.mp3’
In DNS, create a CNAME media CNAME media.thisamericanlife.org.s3.amazonaws.com
Link to your podcast at @http://media.thisamericanlife.org/podcast/332.mp3’

That’s it. To learn more read Scalable Media Hosting with Amazon S3

What about streamguys.com?

Continue to use them for streaming. There no reason you can’t split the hosting for your downloadable podcasts vs. your streaming media.

Downsides

I can’t think of any. There’s no upfront cost, no commitment. You could even use it for a handful of your podcasts to make sure your happy. And give everyone on staff a raise!

RPM building

Peter Burkholder — Thu, 08 May 2008 06:20:00 -0700

It seems that most resources on the Internet about building RPMs are woefully out of date. If you want to unpack and work on RPM source packages in, say your home directory, most documentation points you to settings for your $HOME/.rpmrc file

Then you get errors like this:


error: bad option 'topdir' at /home/burkholp/.rpmrc:1

The use of .rpmrc is obsolete. Instead you’ll want a .rpmmacros file like the following:


%_topdir        %( echo ${HOME}/rpmbuild )
%_tmppath       %{_topdir}/tmp

%packager      Peter Burkholder <pburkholder@pobox.com>

Tornado (but only an EF1)

Peter Burkholder — Thu, 24 Apr 2008 19:18:00 -0700

On Sunday, April 20, our family joined some friends at a local indoor splash pool. We were expecting nothing but rain, rain, and more rain, and needed to get the kids good and worn out.

The boys had a great time, and had finally gotten up the gumption to go through the giant slide that started up near the facility ceiling, did a figure 8 outside, and landed with a splash into the bottom pool. We were waiting in line for our third run when I commented to Shannen that it looked like it was raining buckets outside. Just then we could see the weather going crazy, with sheets of rain slashing horizontally and the trees doing a crazy dance. I shouted to Shannen to take cover, and we were running down the stairs each with one boy when the lifeguards starting blowing on their whistles to clear the pool.

What I thought was just a severe thunderstorm was an EF1 tornado that missed us by 150 feet.

Here’s a link to local TV report on the Maryland weather. Skip the first half of the story to get to the part about the Chillum, Maryland tornado.

VIDEO

NATIONAL WEATHER SERVICE REPORT

MY GOOGLE MAP—I went to the splash park today to map this out. Because we left on Sunday through the front, we didn’t see what had happened out on the back parking lot. If you zoom in, you can see the outside loop of the blue tube slide:

Since we were just a few feet below the roof, I shudder to think what would’ve come of us had it been our roof that came off instead.

Apache and "shared memory" issues on Linux

Peter Burkholder — Thu, 10 Apr 2008 06:07:00 -0700

This morning I was having trouble getting Apache (2.0.X) to restart. I was getting these error messages:

[Thu Apr 10 08:32:49 2008] [crit] (17)File exists: unable to create scoreboard "/var/private/logs/apache_runtime_status" (name-based shared memory failure)

and I kept deleting the offending file, and making sure that permissions all along the path were correct. No dice.

So I restarted Apache again running strace:

strace /usr/sbin/httpd2-prefork -X -Dprivate '-CPidFile /var/run/apache2.private.pid' -f /etc/apache2/httpd.conf

and saw output like this:

unlink("/var/private/logs/apache_runtime_status") = -1 ENOENT (No such file or directory)
open("/var/private/logs/apache_runtime_status", O_WRONLY|O_CREAT|O_EXCL, 0666) = 9
stat("/var/private/logs/apache_runtime_status", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
shmget(16908303, 40824, IPC_CREAT|IPC_EXCL|0600) = -1 EEXIST (File exists)
write(2, "[Thu Apr 10 08:18:46 2008] [crit"..., 168) = 168

Red Herring

Turns out that worrying about the filesystem was a distraction. The real issue is that the previous Apache had left behind shared memory segments that it couldn’t access anymore. Now I won’t pretend that I understand the ins and outs of shared memory on Linux, but a bit of Googling led me, fortunately to Sven Vermeulen’s blog, where he shared a similar experience on Solaris. I would’ve left nice comments on his blog, but as he doesn’t take comments, I’ll give a shout out to him here.

Anyhow, the ipcs and ipcrm commands come to the rescue:

# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x0102000f 99942402   root      600        40824      0

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x00001f58 0          root       600        0            0

Ah-ha—there’s shmid at 99942402. Let’s rm that:

# ipcrm -m 99942402
# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x00001f58 0          root       600        0            0

And apache started right up. Yea.

P.S. One find out more about a shared memory segment with something like: ipcs -m -i 99942402, which will report:

Shared memory Segment shmid=99942402
uid=0   gid=0   cuid=0  cgid=0
mode=0600       access_perms=0600
bytes=40824     lpid=6987       cpid=6987       nattch=0
att_time=Thu Apr 10 08:20:48 2008  
det_time=Thu Apr 10 08:21:55 2008  
change_time=Thu Apr 10 08:20:48 2008

Looking up IPs with mDNS and dns-sd on Mac OS X 10.4

Peter Burkholder — Wed, 02 Jan 2008 18:54:00 -0800

I forgot the IP address of my printer on my local area network at home.

And my Mac quite cleverly only had the mdns name in the DeviceURI, but for setting up an older Windows system I really wanted the IP. Here’s how:

Browse for printers: dns-sd -B _printers._tcp .. :


  Timestamp     A/R Flags if Domain                   Service Type             Instance Name
21:56:43.972  Add     2  7 local.                   _printer._tcp.           Brother HL-2070N series

Lookup the service info for the printer you’ve found: dns-sd -L "Brother HL-2070N series" _printer._tcp:


21:58:30.695  Brother\032HL-2070N\032series._printer._tcp.local. can be reached at BRN-7E6301.local.:515 ...

Query on the domain name: dns-sd -Q BRN-7E6301.local.:


Timestamp     A/R Flags if Name                             T   C Rdata
21:59:52.553  Add     2  7 BRN-7E6301.local.                1   1 192.168.15.103

The reason to bother documenting this? Because the Mac Os X man page for dns-sd doesn’t document the use of the -Q switch.

Grrrrrrrrr.

Processing gzipped files with Perl IO:Uncompress

Peter Burkholder — Wed, 02 Jan 2008 11:23:00 -0800

Idiom of the day:


use IO::Uncompress::Gunzip qw(gunzip $GunzipError) ;

foreach $host ( "foo", "bar", "dev", "teset") {
  foreach $site ( "public", "private" ) {
   foreach $date ( "20071223", "20071224", "20071225", "20071226", "20071227", "20071228", "20071229", "20071230", "20071231", "20080101" ) {

     $input="/web/$host/$site/logs/access-$date.gz";
     next unless -f $input;
     my $z = new IO::Uncompress::Gunzip $input or die "IO::Uncompress::Gunzip failed: $GunzipError\n";

        while (<$z>) {
          next unless / 200 /;
          @F=split;
          ($URI, $therest) = split(/\?/, $F[6], 2);
          # perl regex operation on the URI

        }
   }
  }
}

Set phasers on kill

Peter Burkholder — Fri, 23 Nov 2007 20:39:00 -0800

The first time I ever used a computer was the Friday after Thanksgiving, 1978, when I got to use the DEC PDP 11/70 system on the campus of Goshen College, where my Dad taught. I was immediately entranced by the power and mystery of the digital computer, and quickly hooked on trek

Today, 29 years later, I’ve found that Trek lives on, and I could obtain C-code for Mac OS X that provided an experience akin to my first experience long ago, without having to bike two miles into campus. It took me about 40 minutes to gain enough competence to win my first game by destroying four enemies.