The Final Shift

The above image has always been very poignant for me. That's Avrora and Orest in the Dominican Republic with their mom.  This was in June 2011.  Orest was nearly 2.  Avrora was 6.  I didn't take the picture (I was in the US working); however, I always felt the picture was from my POV.  My daughter ready to walk out into the light.  My son looking out but turned somewhat, wondering if he ought to go back.  Are they moving on, walking away, growing up?  Or am I slipping away into the dark with my daughter accepting and my son not so sure?  Or is this the future,my children are that bright future and my mortality and the present is the dark, narrow tunnel?  Or is the darkness the fear of change and the bright that future?  Or....?  There are a million different thoughts which go with the future.

 The future is now though.  Whatever the meaning of the above, I am facing the future today.  Today is my last working day at NERSC, the supercomputing center I have been at since June 2001.  After 14 years, I am shifting from HPC geek to rocket scientist full time.  I am trading in the Singularity (Rapture of the Nerds!) for the Space Enthusiast (If Only We Had the Will Power!). 

Or perhaps I am just trying to escape Skynet. IN SPACE!

This shift signals the end of my career in supercomputing.  I first logged into a Cray YMP in fall 1990.  Out of the past 25 years, I have had an account on a supercomputer somewhere within DOE with a short stretch from 1998 to 2001 as the exception.  I have worked as a user or admin on Cray XMPs, YMPs, C90s, J90s, T3Es, XT4s, XT5s, XC30s and XE6s.  I have worked on IBM SP3s, SP5s and a mongo iDataPlex.  I have also worked on Thinking Machines CM200 and CM5s.  Even the odd largish SGI Altix, white box linux clusters and even a Linux Networx cluster.  I have managed file systems which both span multiple supercomputers and even one which spanned multiple states.  I've assembled a data transfer system which can packet cannon a sustained 2.5 terabits per second for bandwidth.

I've done a lot.  I've seen a lot.  I've also made it to the point where everything I saw in the beginning of my HPC career is being revisited.  Its not just a silly saying when everything old is new again.  In all the architecture meetings and data handling meetings, this has been almost the uniform theme.  And its uninteresting.  When that happens, its time to go...

Fortunately, for me, I'd laid down the plans pretty thoroughly to move on.

And now I am.

From now on, I am a full time rocket scientist.  Its more than a little scary.

Its a major life change.  I have been working 17 hour days for the past year.  Now I'll shift down to 12 hour days, still long, but it will allow me to be better engaged in many things.  Especially my kids.  For that, I am grateful if nothing else.  However, this is a new business and I AM THE BOSS.  HOLY SHBT.

One of the life changes is that I am going to forgo doing much with the blog.  The rapid updates recently were as much to see if I could get the old backlog of articles I'd saved done before I wound this down.  However, I will not completely abandon the blog.  However, for the most part,  The blog will update with original content only.  It will probably only be weekly rather than daily.

However, I threw up the ads on here (for those of you not using a blocker).  If there are enough clicks for a cup of coffee at a coffee shop per week, I'll guarantee a new article.   In fact, let's turn this around.  I'll write an article on space, HPC, the robopocalypse, Ukraine or a paleo post every time I get a cup o joe paid for.

Actually, let's nuance this a bit.  A cup of coffee is worth a Ukraine update or Robopocalypse report or equivalent.  Cup of coffee and a munchie is worth a xenopermian or equivalent.  A serious research article will require a full meal folks.  :P

Appropriate end music.  

End of a Personal Era

waiting to be disposed

This past week I uninstalled a lot of DDN 9550s.  These were the workhorse storage platforms we used for our global filesystem for years.  Too long, in fact, but budgets are what budgets are and storage normally gets the short end of the stick at HPC centers.  However, the 9550s were really, really stable.  They used 300 GB fiber channel drives to come to a total of somewhere around 720 terabytes of disk.  My loss rate for them was less than 10% - I believe somewhere around 2%, actually - of the newer 9900s with the 1 TB SATA drives.

my row of 9900s

The DDN 9900s are fine, really.  I manage 11 of them with those 1 TB SATA drives (see below).  This comes to about 3.3 petabytes worth of our global filesytems.  Interestingly, the problems which I have are not seen by my compatriots who work with the SAS drives.  However, their SAS drives are 300 GB, so if I were to substitute mine with those drives, while more reliable, they would give a capacity hit down to less than one petabyte. Those are not the newest systems, obviously, but they are useful.  Even if they give me most of my headaches.

my row of sfa12kes, only first 4 racks are mine

These days the new toys are my sfa12kes.  These are embedded servers with virtual machines which run on the disk controllers.  They have been remarkably stable and nonfussy.  I have my issues with them still (specifically their software stack (dude, rolling upgrades, really! need! this!)), but DDN is making progress on its development.  The five 12kes use 3 TB near-line SAS drives which gives me 5.1 petabytes of spinning disk (and I'd like to fill them out to their full capacity for 6 PB, but, again, a budgetary issue.  We spec'ed them for a specific bandwidth/capacity ratio.  Unlike the previous two types of DDN pictures here, not only are the 12kes with embedded servers, they have some spiffy FDR infiniband cards.  Half are tasked to be 10 Gig ethernet, a quarter are FDR IB and the rest of QDR IB (legacy systems).  Plans are afoot to move everything onto FDR IB and use ethernet gateways for the legacy systems.  We have to get rid of the QDR systems though.

 

  seismic isolation planks where the 9550s were

 

At any rate, this is an end of a personal era at NERSC.  No more FC drives for me.   And a glimpse of some of what I do at the dayjob.

Fast Eddy: Our New Supercomputer (at the day job) and some sad news

link.

This is tempered by the fact a coworker died.  He had a medical emergency at work and despite the efforts of folks here went into a coma.  He passed on friday.  I've known him for almost 13 years.  He even interviewed me when I came here and was very kind after a grueling day of interviews.  We worked together on the beginnings of the nersc global file system and then later when he was in the tape systems.  He always had a funny story or gentle, yet snarky comment which would lift my spirits.  He is survived by his wife.

Mike, we will miss you.  And, Mike, I already do.

What a 2 Petaflop (sustained) Supercomputer Looks like Once it is Skinned

The Faces of the Machines of NERSC

Hopper and Edison.  The latter is slowly getting the nickname of Fast Eddie.  Fast Eddie got its first, early users as part of acceptance.  Both are Crays.  I'd put up pix of Carver and the other machines, but IBM doesn't do the nifty art like Cray does for their machines.

Progress on LBNL/NERSC's New Building

before construction via google maps, after site clearing on google maps and a flyover from someone's plane in the beginning of January.  The building's name is Computational Research and Theory.  That's right, CRT.  Oy.  We couldn't get a cool name like the Helios Building (solar power research) or the Molecular Foundry (nanotech).  We got CRT.  Thanks, Chu!

At least it'll be a monster of a computer room and be energy efficient.  Still gonna be fugly though.