Re: Domino Utilities and Server Performance Issues -- DOMINO400

Chris,

Thanks!  This is exactly what we are doing tonight.  And although we have
found a potential fix, we are still scratching our heads as to what caused
this to happen in the first place.  Basically, is there anything that
anyone is aware of that would cause disk performance to significantly
change is such a manner that prior to this weekend, all 22 disk were
between 3 and 9% utilized and after the weekend, 6 disk are utilized at
50-70% while the other 16 remained at the normal level?

Thanks,

Jose del Risco
Corporate Domino Administrator
Oshkosh Truck Corporation

(920) 235.9151  x 2741



                                                                           
             Chris Whisonant                                               
             <Chris.Whisonant@                                             
             comporium.com>                                             To 
             Sent by:                  Lotus Domino on the iSeries / AS400 
             domino400-bounces         <domino400@xxxxxxxxxxxx>            
             +jdelrisco=oshtru                                          cc 
             ck.com@xxxxxxxxxx                                             
             om                                                    Subject 
                                       Re: Domino Utilities and Server     
                                       Performance Issues                  
             01/25/2007 12:58                                              
             PM                                                            
                                                                           
                                                                           
             Please respond to                                             
              Lotus Domino on                                              
               the iSeries /                                               
                   AS400                                                   
             <domino400@midran                                             
                  ge.com>                                                  
                                                                           
                                                                           




Jose,

Well that makes sense! STRASPBAL should help out your performance
significantly!! :)

Chris Whisonant
Senior Mid-Range Systems Administrator
Comporium
803-326-7270
IBM Certified Specialist -- eServer i5 iSeries Domino Technical Solutions
V5R3
IBM eServer Certified Systems Expert - iSeries Technical Solutions V5R2
IBM Certified System Administrator - Lotus Notes and Domino 6/6.5
IBM Certified System Administrator - Lotus Notes and Domino 7
IBM Certified Associate Developer - Lotus Notes and Domino 6/6.5

domino400-bounces+chris.whisonant=comporium.com@xxxxxxxxxxxx wrote on
01/25/2007 01:53:55 PM:

Chris,

I may have stated the information incorrectly. Of the 22 drives, 6
are being utilized at 50-70% while the other 16 are averaging 5%.
These disk store nothing but Domino and the entire box runs nothing
but Domino.
-- Sent from my Blackberry Wireless Device --
Jose L. del Risco
Oshkosh Truck Corporation
Corporate Domino Administrator
(920) 420-5646
Jdelrisco@xxxxxxxxxxxx


----- Original Message -----
From: Chris Whisonant [Chris.Whisonant@xxxxxxxxxxxxx]
Sent: 01/25/2007 01:08 PM
To: Lotus Domino on the iSeries / AS400 <domino400@xxxxxxxxxxxx>
Subject: Re: Domino Utilities and Server Performance Issues



Correct me if I'm wrong, but i5/OS doesn't create data like that. I

would

be REALLY curious to see how this person determined that ALL of the

Domino

data was only on 1/4 of the disks. Are those disks separated in a user
ASP? That could be possible, but you would have had to poing the
/LOTUS/DATA directory to that user ASP...

Chris Whisonant
Senior Mid-Range Systems Administrator
Comporium
803-326-7270
IBM Certified Specialist -- eServer i5 iSeries Domino Technical

Solutions

V5R3
IBM eServer Certified Systems Expert - iSeries Technical Solutions V5R2
IBM Certified System Administrator - Lotus Notes and Domino 6/6.5
IBM Certified System Administrator - Lotus Notes and Domino 7
IBM Certified Associate Developer - Lotus Notes and Domino 6/6.5

domino400-bounces+chris.whisonant=comporium.com@xxxxxxxxxxxx wrote on
01/25/2007 08:24:16 AM:


How much cache is on the controller that controls the drives on the

iSeries

box that hosts Domino?  Domino seems to really perform better with a

hefty

amount of disk controller cache.

Of course that doesn't explain why things were fine, then got bad. I'm

not

sure what was the event that concentrated your Domino specific stuff

on

of the 22 drives... i'd have the guy who found that situation keep

digging

further into it!  He should also be able to talk with IBM support for

some

possible reasons why...


             JDelRisco@oshtruc
             k.com
             Sent by: To
             domino400-bounces         domino400@xxxxxxxxxxxx
             +chadb=wheeling-n cc
             isshin.com@midran
             ge.com Subject
                                       Domino Utilities and Server
                                       Performance Issues
             01/24/2007 06:23
             PM


             Please respond to
              Lotus Domino on
               the iSeries /
                   AS400
             <domino400@midran
                  ge.com>







Hello Everyone,

I have been experiencing Lotus Notes performance issues since Monday
morning.  At first we thought the problem to be a run away process (

Router

task on a server was running at 80-90% for about 35min and it wasn't

doing

anything ).  Since then ( Monday at 10:35am ), CPU for the entire LPAR

has

been averaging 30% and peaking at 65%.  Still, as of the time of this
email, we are seeing intermittent performance issues. And by that I

mean,

everything seems to run normally then for some seemingly random

reason,

it

take upwards of 4 minutes just to open an email that does not have an
attachment or switch to a different folder view, etc.  We sniffed the
network and found no network issues occurring during these moments

that

Lotus Notes runs slow.  These moments tend to last about 5-10 minutes

and

all goes back to normal.  The performance problems to not seem to tie

into

any server task or event taking place.

After eliminating CPU we checked Memory.  The iSeries guru at our
organization informed me that there didn't appear to be any memory

issues

but did notice that 6 of our disks ( of 22 ) were being hit hard. When

he

did some further digging he noticed that ALL of the data for all 10

server

instances on our LPAR were all on 6 of the 22 disks that we have. This

was

not the case last week.  It would seem that the performance issues

were

related to disk and he is performing a task to evenly distribute the

data

among the 22 disks by usage.  Now you are all probably wondering, "How

did

all the data end up on just 6 disks when it was spread across 22 last
week?"......  That, my friends, is the $100,000 question.  If anyone

has

any ideas I would love to hear them.  Some background info.

Saturday night we applied PTF's to the LPAR.
We also ran Fixup on 3 of our server, and Updall -r  on 2 of the 3

servers

that Fixup was run on.

That is the extent of the changes that occurred this weekend.  I have

directors meeting in the morning at which they will be asking me why

Lotus

Notes is running slow.  I believe I have determined what was the

problem

but not the root cause.  If anyone can send me idea before 9am

tomorrow.

Feel free to ask me any questions, I will provide as much detail as I

can.


Thanks,

Jose del Risco
Corporate Domino Administrator
Oshkosh Truck Corporation

(920) 235.9151  x 2741


Although this e-mail and any attachments are believed to be free of

any

virus or other defect which might affect any computer system, it is

the

responsibility of the recipient to check that it is virus-free and the
sender accepts no responsibility or liability for any loss, injury,

damage,

cost or expense arising in any way from receipt or use thereof by the
recipient.

The information contained in this electronic mail message is

confidential

information and intended only for the use of the individual or entity

named

above, and may be privileged.  If the reader of this message is not

the

intended recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is strictly prohibited.

If

you have received this transmission in error, please  contact the

sender

immediately, delete this material from your computer and destroy all
related paper media.  Please note that the documents transmitted are

not

intended to be binding until a hard copy has been manually signed by

all

parties.
Thank you.
_______________________________________________
This is the Lotus Domino on the iSeries / AS400 (Domino400) mailing

list

To post a message email: Domino400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/domino400
or email: Domino400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/domino400.

_____________________________________________________________________________


Scanned by IBM Email Security Management Services powered by

MessageLabs.

For more information please visit http://www.ers.ibm.com

_____________________________________________________________________________

ForwardSourceID:NT00060E3A_______________________________________________

This is the Lotus Domino on the iSeries / AS400 (Domino400) mailing

list

To post a message email: Domino400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/domino400
or email: Domino400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/domino400.



NOTICE: This message contains information which may be confidential. If
the reader is neither the intended recipient nor a person responsible

for

delivering the message to the intended recipient, you are notified that
any distribution, copying or retention of this communication is strictly

prohibited. If you have received this communication in error, please
notify us immediately by return e-mail, telephone or facsimile. Thank

you.

_______________________________________________
This is the Lotus Domino on the iSeries / AS400 (Domino400) mailing list
To post a message email: Domino400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/domino400
or email: Domino400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/domino400.


Although this e-mail and any attachments are believed to be free of
any virus or other defect which might affect any computer system, it
is the responsibility of the recipient to check that it is virus-
free and the sender accepts no responsibility or liability for any
loss, injury, damage, cost or expense arising in any way from
receipt or use thereof by the recipient.

The information contained in this electronic mail message is
confidential information and intended only for the use of the
individual or entity named above, and may be privileged.  If the
reader of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited.  If you have received this
transmission in error, please  contact the sender immediately,
delete this material from your computer and destroy all related
paper media.  Please note that the documents transmitted are not
intended to be binding until a hard copy has been manually signed by
all parties.
Thank you.

_______________________________________________
This is the Lotus Domino on the iSeries / AS400 (Domino400) mailing list
To post a message email: Domino400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/domino400
or email: Domino400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/domino400.



NOTICE: This message contains information which may be confidential. If
the reader is neither the intended recipient nor a person responsible for
delivering the message to the intended recipient, you are notified that
any distribution, copying or retention of this communication is strictly
prohibited. If you have received this communication in error, please
notify us immediately by return e-mail, telephone or facsimile. Thank you.
_______________________________________________
This is the Lotus Domino on the iSeries / AS400 (Domino400) mailing list
To post a message email: Domino400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/domino400
or email: Domino400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/domino400.




Although this e-mail and any attachments are believed to be free of any virus 
or other defect which might affect any computer system, it is the 
responsibility of the recipient to check that it is virus-free and the sender 
accepts no responsibility or liability for any loss, injury, damage, cost or 
expense arising in any way from receipt or use thereof by the recipient.

The information contained in this electronic mail message is confidential 
information and intended only for the use of the individual or entity named 
above, and may be privileged.  If the reader of this message is not the 
intended recipient, you are hereby notified that any dissemination, 
distribution or copying of this communication is strictly prohibited.  If you 
have received this transmission in error, please  contact the sender 
immediately, delete this material from your computer and destroy all related 
paper media.  Please note that the documents transmitted are not intended to be 
binding until a hard copy has been manually signed by all parties.
Thank you.