Hi,
In case anyone is still interested on the whole "i on Blade" topic, i
wrote about saving (which, is usually quite crucial).
Link:
http://projectdream.org/wordpress/2008/07/21/ibm-i-on-blade-how-to-save/
Again, i don't know how much longer i'll be able to keep this test
hardware around. If you have any questions or want me to test
something for you, just drop me a mail.
And full text:
Setting up the hardware
Finally, after almost two months i was able to get my hands on a SAS
cable with the correct pinout to connect the TS3100 library to the SAS
Connectivity Module. After plugging in the cable, both link lights
went up. The TS3100 was connected - physically at least.
I logged into the Storage Configuration Manager, and dared to assign
the HS21 Windows Blade and the JS12 Blade to the external SAS Port.
When looking at the Windows Device Manager, it immediately recognized
the Tape Drive, but didn't recognize the media changer. No bummer, i
haven't installed BackupExec yet.
It's important to notice that the IBM i OS will never, ever see your
tape drive. You cannot connect the tape drive to the IBM i OS, only to
VIOS. This means that you always will do a disk to disk backup to the
VIOS partition, and then use VIOS to save the D2D Image to tape. This
is not optimal, as this means that you cannot use BRMS to manage media
(you can still use it for saving, though). It also adds another layer
of indirection that makes automating backups more difficult. Another
important point is that other i machines will not be able to read the
VIOS created tapes, and that your i Blade will not be able to read
tapes created by standalone POWER machines running IBM i.
Configuring VIOS
So i went to logon VIOS, running on the JS12 blade. I ran "lsdev |
grep rmt0″, but apparently there was no tape drive to be seen. I ran
"cfgdev", to let the operating system configure devices, but that
wasn't met with success either.
This time, i chose the easy way out. I just rebooted the entire blade,
and finally, the tapedrive showed up:
$ lsdev -dev rmt0 -attr
attribute value description
user_settable
block_size 262144 BLOCK size (0=variable length)
True
delay 45 Set delay after a FAILED command
True
density_set_1 0 DENSITY setting #1
True
density_set_2 0 DENSITY setting #2
True
extfm yes Use EXTENDED file marks
True
mode yes Use DEVICE BUFFERS during writes
True
res_support no RESERVE/RELEASE support
True
ret_error no RETURN error on tape change or
reset True
rwtimeout 144 Set timeout for the READ or WRITE
command True
var_block_size 0 BLOCK SIZE for variable length
support True
ww_id 5000e1111c878001 World Wide Identifier
False
Okay. So as the TS3100 is a LTO4 Tape Library, the next step obviously
was to load a tape into the tape drive. Now this can be done through
the Web Interface or the Control Panel of the library, but that's not
the way you want to go during day to day backup - media moving should
be handled by the backup software (called BRMS on IBM i).
But alas, this is not as easy on VIOS. Basically, you can get an AIX
root shell on VIOS by typing "oem_setup_env" - and then there are
several AIX commands to manage a tape library.
There was only one problem:
# mtlib
ksh: mtlib: not found.
# tapeutil
ksh: tapeutil: not found.
They're not there. A quick google search revealed nothing. I didn't
know how IBM thought how we should use a tape library. In sequential
mode? Or is there some way to manage a tape library in VIOS? If you
know, tell me!
So, the next step was to create writable optical media, so i could
start with creating a Save 21 of the system. This seemed as a sane
first step.
Initializing the media on the i side
First, i needed to create a virtual optical volume in a volume library
on VIOS - i already had the volume library from the IBM i
installation, so all i needed to do was to add a writable optical
volume. The system used about 40GB DASD, so i created a virtual
optical device with a size of 80GB to leave room for future growth.
This process took around 2 hours, probably because VIOS pre-blanked
the file (it effectively used up 80GB on disk). A rate of 0.6GB/min.
During this operation, the VIOS webinterface grinded to halt (didn't
respond), but SSH was still available and responded very slowly to
each command. I suppose this is some issue with the disk controller,
either the driver or firmware.
According to vmstat, most of the time is spent in system or IO wait context:
kthr memory page faults cpu
----- ----------- ------------------------ ------------
-----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id
wa pc ec
1 1 203749 6107 0 0 0 2485 3279 0 150 29483 6542 6 9
77 7 0.18 17.5
1 1 203732 6167 0 0 0 2464 2869 0 153 30479 6333 7 9
74 10 0.18 17.6
The next step was to load the newly created virtual media into the
virtual optical drive that was attached to the IBM i partition. This
was rather easy to do, just click through the web interface.
Now, we need to initialize the optical volume in IBM i OS:
INZOPT NEWVOL('Save21′)
DEV(OPT01)
CHECK(*NO)
TYPE(*PRIMARY)
This finished rather quickly, and i then started the Save 21.
The Save 21
GO SAVE/21. The performance wasn't much better, though. the vmstat
looked the same as above, indicating the same problem.
Further debugging using iostat revealed that a volume group is not a
RAID1 array - but nonetheless, the disk subsystem is behaving oddly:
hdisk0, hdisk1: VIOS installation - part of it is mirrored VIOS,
other part is volume group that is spanned across these two disks
hdisk2-5: SCSI passthrough to the IBM i
tty: tin tout avg-cpu: % user % sys % idle %
iowait physc % entc
0.0 614.0 0.0 7.7 72.2
20.1 0.1 8.3
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 1.0 8.0 2.0 0 8
hdisk1 85.0 20036.0 179.0 10080 9956
hdisk2 4.0 926.0 50.0 922 4
hdisk3 14.0 4481.0 98.0 4477 4
hdisk4 13.0 3730.0 96.0 3726 4
hdisk5 3.0 652.0 36.0 648 4
This is pure sequential IO - why is it reading from the disk? A
similar picture was seen throughout the whole backup - even when
backing up image catalogs from the IFS. hdisk1 consistently showed
strong write activity. No idea why.
My AIX skills are weak, and i didn't know a way to see on which files
the write IO happened - however it's important to know that read and
write always showed the same numbers. To me, this looks like a problem
- either in my config, firmware levels, or even a problem on IBMs
side.
Either way, the Save 21 completed in 45 minutes. At around 40GB, this
brings us to 0.9GB/min.
Backing up the virtual image to tape
The next step is to backup to our LTO4 tape.
Here's how the backup itself looks:
# find /var/vio/VMLibrary/D2D_1 -print | backup -ivqf /dev/rmt0 -b 512
Backing up to /dev/rmt0.
Cluster 262144 bytes (512 blocks).
Volume 1 on /dev/rmt0
Backup finished on Mon Jul 21 20:45:18 CEST 2008; there are
167772672 blocks on 1 volumes.
And the sequential IO performance is much more reasonable:
tty: tin tout avg-cpu: % user % sys % idle %
iowait physc % entc
0.0 615.0 11.2 29.0 58.2
1.6 0.4 40.9
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 99.0 81920.0 160.0 81920 0
hdisk2 17.0 2250.0 29.0 0 2250
hdisk3 12.0 2236.0 28.0 0 2236
hdisk4 12.0 2385.0 30.0 0 2385
hdisk5 16.0 2250.0 29.0 0 2250
That's roughly 5GB per Minute. A very decent performance.
Interoperability
Now, this is where it gets interesting. Attached to the BladeCenter S,
we have a TS3100 with a single drive, in the BladeCenter S we have
three Intel Blades running Windows Server 2008 and one POWER Blade
running IBM i.
We need to back up all this to the TS3100 - on the Windows Side, i'll
be using BackupExec 12, on the i Side VIOS. How do i make sure that
the tape drive can be used from both sides, without to much
interaction?
The SAS Connectivity Module can attach the same port to multiple
Blades. So installed BackupExec on one of the Windows Blades, just to
see how interoperability would work out.
I ran a backup & restore on another tape, from BackupExec. This worked
fine. The next step was loading the tape from the i Save back, and
then run a test restore from that. Unfortunately, i couldn't use
BackupExec to just move the tape in the drive, so i had to use the
TS3100 Web Interface again.
I looked at the tape drive from VIOS, which also seemed okay. I also
saw how much space was used on the VIOS tape after checking it up with
BackupExec (this is stored on a small RFID Chip on the Tape itself).
But the real test was yet to come:
Restoring from Tape
I started the restore from tape.
# restore -xvqf /dev/rmt0 -b 512 /var/vio/VMLibrary/D2D_1
New volume on /dev/rmt0:
Cluster size is 262144 bytes (512 blocks).
The volume number is 1.
The backup date is: Mon Jul 21 20:24:04 CEST 2008
Files are backed up by name.
The user is root.
x 85899345920 /var/vio/VMLibrary/D2D_1
The total size is 85899345920 bytes.
The number of restored files is 1.
iostat also told me that the performance on restore was bit worse
than when backing up:
tty: tin tout avg-cpu: % user % sys % idle %
iowait physc % entc
0.0 611.0 14.7 85.0 0.0
0.2 1.0 101.9
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 100.0 48216.0 68.0 0 48216
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 0.0 0.0 0.0 0 0
So after having restored the image file to VIOS, i could IPL from it
and run a complete system restore. Nice. I didn't want to scratch my
whole setup and wait until a full restore, so i tried something
simpler.
In the end, it turned out to be around 2GB/min. This was a lot faster
than the creation of the file, which seems really odd to me.
Running a Test restore on the IBM i side
After restoring the file in the VMLibrary, it automatically appeared
again in VIOS. I only had to mount it to my IBM i partition. This
could easily be done through the VIOS web interface.
I ran a simple restore:
RSTLIB SAVLIB(AVNEDIAS) DEV(OPT01)
With simple results:
12 Objekt(e) von AVNEDIAS nach AVNEDIAS zurückgespeichert.
I also tried IPLing from the virtual optical media, which brought me
into the limited paging DST. Nice!
Is this good?
AS you can see, this is not your fathers AS/400. This is a POWER Blade
running IBM i. You'll need to learn a bit about VIOS and AIX in order
to make any sense on how this whole stuff works. But it's not rocket
science - i only know a bit about Linux, and was able to figure out
the tasks i needed to do.
But now, how should one run this in production?
The current configuration i have seems unsuitable to production to me.
* VIOS/AIX can't handle the tape library in random mode. This is a
big letdown.
* Backing up to tape and restoring seems very, umm, basic to me
* The fact that you are using virtual optical devices, with no
ability on the i side to change media, makes a usuable backup
procedure hard to implement
* Automation on the VIOS side could be implemented by the i
running the ssh client in command execution mode (similar to how this
is used with the HMC
* Integration between the Windows and VIOS seems cumbersome
Solutions
Two Half-Height Tapedrives in a partitioned library
The TS3100 can be partitioned, and we could install two half-height
LTO4 tapes. Reserving 12 Slots for Windows in Random Mode, and 12
Slots for VIOS in Sequential Mode. This would work, but there's one
huge downside: Price
Backing up on Windows only
Instead of using VIOS and clumsy hacks to get a halfway decent
functionality, you can use Windows to backup everything. Save 21 and
Systems Saves would run through VIOS in order to create bootable
media, and then retrieved on the Windows side using SFTP.
For daily backups, we can use savefiles directly on the i, which is
probably easier to deal with for most IBM i admins. These can be
retrieved from the Windows side using FTP/TLS. The downside: If you
have i and Windows people that work well together, i don't see much of
an issue. But if not, you got a big mess in your hands.
SAS Passthrough to IBM i
Unfortunately, this does option not exist yet. This is something that
IBM should work on intensively. It will allow i admins to use well
established backup processes with full library integration using BRMS.
Conclusions
Backing up the IBM i on Blade isn't exactly easier than backing up a
standalone POWER machine. In fact, it's more difficult and requires
additional skills.
Before buying a JS12 blade running IBM i, make sure that you think
your disaster recovery strategy through completely. Your business
partner should be able to help you with this.
Planning is crucial - Backup & Restore on the blade is different, and
you'll need to deal with VIOS when creating a procedure for fully
automated Save 21 backups.
midrange.com
