Re: Horror Stories

[Larry Bolhuis <lbolhuis@xxxxxxxxxx>]

Some years ago I managed an AS/400 shop in the pre-RAID days. I left,
but about 6 years later got called to help them when the manager quit.
First thing I checked was backup status. Seemed they hadn't done a
Save-21 (or anything at all besides nightly saves) since I left, so we
wrote out the instructions and went over them with the operator for a
Full Save the next night. The next week we powered down the machine to
add an ethernet card, upon power up, one of the drives failed.  Pretty
amazing, we figure, that we had a save-21 from just days prior, so we
got the properly labeled and dated tape from the vault. While the CE is
doing his drive thing the operator sees the tape and asks: "You really
didn't expect to do anything with that did you?" "Of Course!" we replied
in unison to which she added: "Well it's blank because the backup
wouldn't work." Picture two 6'4" guys ready to dismember a 5'2"
operator. If she didn't fear for her life at that second it was only
through stupidity or that we looked so silly with our jaws laying on the
floor. Several explitives (that would be deleted anyway) later we
invited her to leave while she still could. Turns out she was on side
two of a two session console but had never asked for help or even told 
her supervisor that the backup had not been done. She just put the tape
in the vault!!!  We spent 4 solid 24 hour days rebuilding that machine
from 6 year old backups and nightlies. To make matters worse the
Save-21s they DID have were on tape they could no longer read so we had
to translate the buggers before we could start! Some stuff was never
recovered because besides not doing Save 21s there were new libraries
added to the system that were not added to the nightly backups. Add to
that they had removed things like SAVCFG and SAVSECDTA from the backup
because they had run out of tape (That stuff doesn't change in 6 years,
does it?)  The only saving grace was they hadn't applied an OS Upgrade
or a PTF in the 6 years either!!

Probably for good reason, that company is bankrupt now.

 - Larry

-- 
Larry Bolhuis           | Cogito ergo mercari iSeries 
Arbor Solutions, Inc.   |
(616) 451-2500          |               (I think, therefore I buy iSeries.)
(616) 451-2571 -fax     |
lbolhuis@arbsol.com     |        #3  1951-2001
+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---