Wampum

In a recent WaPo there is Ruth Marcus The Family as Firing Offense: For Too Many Workers, Emergencies at Home Force Stark Choices. I'm going to take off the veil. You are reading the blog of someone fired by the school nurse.

The week prior to my departure for DisneyWorld I produced data for six and ten shelves of server blades servicing 100 clients, for Panasas' propriatary filesystem, and for five and twelve shelves of server blades and 100 clients, for Sun's open Network File System, for Aramco/ECC. My manager was all smiles and things-are-great-enjoy-your-vacation on the 24th of February, the LANL project would begin on my return.

On our way back from DisneyWorld to Pittsburgh we encountered an emergency on the road, and a stark choice. In the middle of the day I was due back at Panasas I finally knew when I'd be back in Pittsburgh, and called my manager. He wasn't in, so I left a message.

The next morning my manager had clenched teeth and you-are-unprofessional-in-spades message. He'd approved a return date of March 7th. It was March 8th. He made a point of saying that he wasn't going to change the schedule because I'd returned to Panasas a day latter than scheduled. I thought that was an interesting point, but as he'd mentioned he'd OCD at some prior point, I put that down to illness, not project plan constraints revealed by a GANTT or a PERT chart, or real experience.

LANL had funded the development of RAID1+0 semantics to PanFS, the propriatary filesystem of Panasas, a parallel filesystem vendor. Client nodes consisting of Linux boxes, numbered in the tens of hundreds, would image process some segment of a wicked large array of data, compute the parity bits (meta-data) for the segment, and send that to server blades numbering in low hundreds, avoiding the head-of-line-like blocking property of systems that send all meta-data, and data, through single, or small hierarchies of filer heads.

I interviewed at Panasas for work on a parallel version of NFS, aka "pNFS", within the IETF's NFSv4 WG. I was hired to do performance, which broadly means to understand the first principles of everything, and rigorously measure evolving snapshots, to identify wins and losses in I/O performance. Two operating systems (three if both the 2.4 and the 2.6 Linux kernels are considered as distinct targets, which they are), and two file systems, and three RAID semantics, and several distinct applications -- weapons ignition sequence imaging, exploratory geophysics imaging, meteorological imaging. All computational fluid dynamics, with domain-specific kinks.

My part of the LANL RAID1+0 project was testing the change to how the MPI library bits were delivered, and as that was the primary tool for measuring how fast the filesystem actually is, the intentional breakage (scheduled new bits), coupled with the delivery of new RAID semantics (RAID1+0), was not, in my opinion, least-risk project planning. I also had to determin the actual performance of the new RAID semantics. Using intentionally broken tools. I didn't see any point in mentioning that I'd worked on the specification of MPIv2, and my name is somewhere in the acks.

By March 10th I'd found that there was a problem. I could get the system to hang. Realm managers lost contact with OSDs, there was thread starvation. It got worse as the days went by. The pioneer-panfs-client-black-raid10 branch was becomming less and less stable as fix for problem created by a prior fix accumulated. Worse, not only were the RAID1+0 numbers degraded, the RAID5 and RAID0 numbers were as well.

Over the weekend of March 18/19 (I was working about 2x "40 hours/week") I decided to see if RAID0 still worked for the standard case, a stripe_unit size of 65536. I found wicked long response times on the OSDs, and wicked low throughput. It was a project catastrophy, unimproved by the absence of any project management I'd seen post-March 8th.

I decided to stop killing myself on March 19th, and go back to somewhere between 9 and 10 hours a day, not from 6am to 9am remote, 9am to 6pm on-site, and 6pm to 11pm remote. I started debugging a memory corruption bug in the LANL version of MPICH, the core of the product test metric, which only ran correctly on the 64bit 2.6 and 32bit 2.4 Linux kernels if compiled on the 32bit 2.4 Linux kernel. By March 23rd I'd got the failure down to a simple case, which still manifested in multiple failures -- I'd quite a collection of stack traces and test cases. The second use of the communications buffer, by one of two (or vastly more) processes, when Steve Moyer poked his head in my office to tell me I was fired. He had a nice letter for me to sign, to the effect that I was dumb as dirt and a whole bunch of meetings where he'd artfully worked to illuminate me, none of which actually transpired, except in his private universe, was proof the firing wasn't arbitrary and capricious. An aggravating circumstance was that I co-authored a draft for the IETF's NFSv4 WG, without prior management authorization (recall, that was why they asked me to come to Pittsburgh in the first place).

The HR creature bobbleheaded along. She wasn't going to authorize my being paid unless I signed their work of epistalary creativity. In fact, she was pretty sure she could keep me in her office until she'd exhausted her repitoire of HR science by the sheer force of her will.

I just put my remaining papers in my bag and left. The root cause was the non-emergency minutia of care for children with autism. And a corporate culture the subject of Ruth Marcus' essay in the WaPo. As for the LANL schedule, it was dead on departure. The plan hadn't anticipated any fundamental surprises, like failure. You may not see the difference between Panasas and its competitiors in the national laboratory nuclear device initiation sequence simulation imaging, or in Aramco/EEC's oil exploration, but you can see it in the Disney vs Pixar imaging. Disney uses Panasas to render images on, and Pixar doesn't. There are other reasons to prefer Pixar's product over Disney's.

Dwight says "One broken alternator away from living under a bridge" to encapsulate the sense of the fragility of almost ordinary life for the working class, and middle class too, here in paradise.

And that's how I see the occasional seasonal, and recurring seasonal, and multi-year resident Mexican migratory worker policy. It is the policy of the United States, it is the policy of the state of California, it is the policy of the state of Arizona, it is the policy of the state of New Mexico, it is the policy of the state of Texas, it is the policy of the state of ... that a burnt-out tail light, a work-place injury, a sick kid, any casual hazard of life, means disproportionate consequences.

The loss of children, the loss of housing, the loss of employment, the loss of freedom to choose -- Mexico or el Norte. Everyone who's ever looked across the table at a manager using time and trivia as a weapon knows how every undocumented person feels in a universe of sadists using documentation as a weapon.