WEDNESDAY

Full stack introspection

Theo Schlosssnagle (OmniTI Laurel Md), jesus at omniIT

Author of “Scalable Internet Architectures”

Whole stack introspection

• MacOSX, BSD, Solaris
• Paul Fox working on Linux port. Not in kernel, so CDDL/GPL, tension not a worry

DTrace approaches

1. Start w/ User
2. Sorcery (not recommended) Twiddle to fix, then verify. Starts w/ Hypotheses
3. Experienced. Starts in middle

• It’s very important to know your optimum operation before you can diagnose anomalous behavior.

SysTap on Linux is way, way behind. Requires righting code and sticking in as module—not a way to work on production boxes.

MacOsX is very well instrumented.

Then follows a masterful demonstration of real-time system diagnoses. Obviously this is not something one learns in an afternoon, but certainly is more fruitful (and probably no harder) than what I hack up in grep, perl, or what not to figure our what’s going on in VKS, or ps, or logfiles.

Take aways:

• Dtrace for Linux may be in the offing. Dtrace awesome. Watch for it.
• I must better understand our architecture from ACE to backends.

Groovy vs. JRuby

A very good, well-organized talk. Both environments are ready to use. Groovy goes to Jars. Performance used to be 20% – 90% of pure Java, but rapidly closing. JRuby may make Java integration in the common cases but Groovy handles the edge cases better. Groovy language still a little immature, e.g. Regexs are not transitive. Groovy allows typing (which can be handy), but JRuby does clean duck-typing. JRuby has all Ruby libraries. Rails works in J2EE stack.

CouchDB.

Freakin’ awesome. Written in Erlang (BTW, Ericksson’s switches running erlang advertize 9 9s (0.3 s annually). I’ll need to write more about it later.

OpenSource Virtualization:

Completely lame presentation. One needs to learn that cutesy slides will emphasize, not obscure, your inexperience.

VmWare ESXi will be free with a non-deprecated API, announced June 22.

VirtualBox

• Maps to VmWare Workstation, Player, Fusion, Server
• Tools -> guest additions
• vmx -> xml
• has Seamless (Unity) latyer
• harder: interfaces, network, disk, audio, USB.
• very similar across OS’s, Solaris Host
• Can run headless and over RDP.

Xen. Etc., but I was getting too annoyed by the talk so I caught the rest of the lightning talks and Google open source update.

Google Open Source Update

Brian DiBona and TKTK

• Melange
• Google Highly open participation contest ==
• GHOP
• DROP

Bofs and Parties

System Administration best practices BOF. Nothing new learned, but maybe I made some contributions.

Met a friendly pair of brothers from Baltimore. Mark and Curt Tilmes. Mark works at UMD (five weeks of vacation a year sounds pretty sweet). We went down to the OpenSolaris party at the DoubleTree, which was too redolent of an overamped frat party. The MindTouch people are having a great party up on the top floor the Red Lion, with a tight live band, nice atmosphere and a sweeping view of Portland across the Willamette. Too bad I’m too tired to stay here long and that it’s not better attended.

THURSDAY

New York Times intro to AWS

Brian Gottfried

• Really needed PDFs from 1851-1981
• Dynamically generate or… pregenerate 11 million PDFs and store statically.
• Copy source to S3 store PDFs in S3 (which has been reliable except for last week)
• 4.3Tb of 20M files
• Parallelization. Into pdfs, so map not reduce.
• Non Hadoop part: read metadata, load TIFsS, scale arrange, generate PDF, write PDF
• Minimal use of HDFS
• Took previous code and conv. to S3 thanks to JetS3S

MapReduce

• Map takes a Key/Value pair and output Key/Value pair
• Reduce takes key & list of assoc. values and combines

HDFS (Yahoo! inc)

Cool, scary. Supports mapreduce. Need to read up.

Pushmi for Subversion

Subversion replication. Probably not relevant to me with Git coming onto our scene.

Configuration management

Commerical products are archaic and monolithic *Tivoli, CA Unicenter

• Help desk, ticketing, asset management, a lot of things badly.

Puppet bcfg2, pikt, cfengine,

Measuring success.

• Availability
• Reduction of incidnets
• Rate of failed change
• Build time and speed to market
• Compliance to SOE, policy or standard

NewStuff:

• Puppetshow A Gui for history and nodes
• iClassify (OS)
• Puppetview Logger (simple PHP app)

Subnode (VMs and containers)

• Zones look like resources
• insides zone looks like a node (but behave differently

Other stuff: (see BOF below)

• webrick – not for more 25 hots. 6000 hosts w/ Mongrel. Scale like any https. w/ loadbalancer.
• puppet query against DB or LDAP.

Supervisor

Python. Built on medusa. Comparable: deamontools, launchd, runit but built for control of customer processes not pid 0.

• supervisord forks arbitrary programs and child (including apache—which you run in FG since it BGs itself).
• supervisorctl on same machine or over xmlrpc.
• ini config file
• captures child stdin/stdout and logs and rotates
• users cannot stop/start arbitrary processes, only predefined processes
• open or by authentication.
• xmlrpc for your own supervisorctl.
• single-process, thread. Your processes cannot block.
• event-notification system
  • in any language
  • can spawn fastcgi processes
• Can supervisor replace our bad sudo moderated scripts ??

Puppet BOF

• TKTK from U. Georgia Board of Regents
• TKTK from EngineYard
• Luke Kanies
• James Turnbull from Australian bank (today’s presenter)
• TKTK from ????
• Me

Puppet is in Git now. Puppet purge nop reports on all aspects of system that aren’t managed. Ralsh takes purge report and generates manifests. Scale by having one puppetmasterd dedicated to CA work and put the others behind LB as fileservers and manifest delivery. Stanford’s configuration takes 30s to parse about 1m to implement.

Svn pre-hook scripts can run parse check. Git lets you run pre-hook on local devel repository and again on core repository

Puppet under pressure to use XMPP instead of HTTP. New release reduces time by 70% by using YAML instead of Marshall.

Automateit Lightweight config management?

RALSH resource abstraction layer shell

FRIDAY

Keynotes

Mostly ho-hum. Microsoft’s opensource VP walked into the lion’s den and emerged mostly unscathed.

Sun’s TKTK on today’s languages. Reference to langpop.com language ranking based on. O’reilly book sales heat map. Tiobe index.

"I'd rather drive nails into my head" than code in PHP. (I didn't know they had over 5000 entries in their global namespace). Languages to watch: Groovy, OCaml (fast!), Erlang (stable, concurrent),  Scala (see Twitter), Fan.

Terms to look up: “Monkey Patching”

Java isn’t going away because the JVM and APIs are so mature, so the environment is being extended to Ruby, Python, Groovy, etc.

NASA

Curt Tilmes at Goddard

Data: Growth 3.2 tB/day; Archive 4.9PB; daily distribution processing 4.2 Tb

MODIS 250m resolution Level 1, raw Level 2, gridded Level 3, best 8 day w/o clouds if possible

...

http:/www.opensource.org/licenses/nasa1.3.php http://opensource.sfc.nasa.gov (Goddard specific, but other centers similar)

Curt gave a nice overview of opensource utilization at NASA and their attempts to contribute back. I would have liked more specifics on his struggle just to release a CPAN module. No specific technology takeaways, but I would like to see NIH/NCBI make a more determined effort to contribute on our periphery activities.

RT3/4

I tackled Jesse Vincent. between sessions. Told him about my RT to JIRA project and my reluctance about it.

RT4 is coming out but is going to be a deep rewrite with Jifty, new DB schema, AJAXy and drag-drop workflow scripting generation. No release date set.

rt3.8 has an ‘extract header’ extension for linking into other ticket systems, which could be useful to me.

REST apis have been growing and are supported. Not XML based, can be used from command line.

Base level support is $5k. Gold is $35k. But is based on ticket numbers.

Open Source Electronics

Tom talked about using open source tools to develop your own open-source hardward projects. Recommended references:

- Art of Electronics - OpAmps for Everyone - MIT OpenCourseware

OpenSource CAD. Start with Paper. Software: gEDA/gaf + PCB (first one I picked). Should be easy.

gschem gsch2pcb to circuit baord

PDB printing is clean fast cheap, credit card. www.4pcb.com www.sunstone.com, Alberta printed fast and cheap only two layer

Circuit simulation: SPICE. closed source freeware: LTSpice. NGspice part of gEDA project.

Digital simulation: Verilog.

Failures

Therac massive radiation overdose. Removed hardware interlocks. Killed six people. ETc. with space program, electric utilities, and so on.

Main points:

• test
• test before launch
• test end to end
• listen to your sysadmins

Buzzwords

• open authentication
• xmpp
• jbod
• json
• erlang