Re: Web site hangs -- DOMINO400

This is as much for me as for anyone else! I compiled this and put it on 
my blog as well at 
http://cwhisonant.blogspot.com/2005/07/debugging-for-domino.html

Debugging for Domino 
Before proceeding, let me say that starting debugging should only be done 
per instructions from IBM. =) Also note that debugging can, by it's very 
nature, impact system performance. Although, I have not noticed any 
performance impacts when I have enabled debugging. This post will only 
deal with configuring the server for debug mode and not with the analysis 
of debug data. Many of these commands are also useful for the Notes Client
.

Let's begin with some of the DEBUG_* options by describing their usage 
along with some information on how to enable them.

1.) DEBUG_THREADID=1
Per this Lotus technote, "This prefixes the console output with the 
process and threadid information in the format [ProcessID:Virtual Thread 
ID-Native Thread ID]. This can be helpful in identifying the process or 
thread holding a semaphore."

2.) DEBUG_NTI_DRVR=1 I cannot find any information on this via technotes. 
My understanding is that this will enable debugging of the TCP/IP NTI 
Driver.

3.) DEBUG_TCP_ALL=1 This turns on debugging for all TCP functions.

4.) DEBUG_TCP_ERRORS=1 This turns on debugging for TCP errors.

5.) DEBUG_DBSERVER=1 This turns on debugging for the DB Server calls that 
may be made.

6.) DEBUG_SHOW_TIMEOUT=1 This turns on debugging for semaphores. See this 
Lotus Technote for information on debugging. See also this Lotus Technote 
for general information on semaphores.

7.) DEBUG_CAPTURE_TIMEOUT=10 This adds timestamps to the semaphore 
debugging.

8.) Debug_outfile=debug.txt This option specifies the location of the 
debug file. As a side note, I have noticed that after a couple of days on 
my iSeries that this file stops growing at 4GB. Lotus could not confirm 
this, but it appears that at 4GB this file becomes circular just like 
transaction logging. The following information is from the Lotus 
Administrator help database:

>From my understanding, these are all dynamic options (except for #8) - 
meaning that you can type "set config DEBUG_THREADID=1" (or any of the 
others) at the console and begin seeing this output. If you have the debug 
outfile or console logging active (see below), then the will be reflected 
in those files as well. Please see this Lotus Technote for more 
information on setting this dynamically.

Syntax: Debug_Outfile=filename
Description: Specifies the file name for the Console Log file. If both 
this setting and the LogFile_Dir setting exist and Debug_Outfile contains 
a fully qualified path name, then LogFile_Dir is not used.
If only the Debug_Outfile setting exists and it contains only a file name, 
then the default path \DATADIRECTORY\IBM_TECHNICAL_SUPPORT is used. If 
neither Debug_Outfile or LogFile_Dir exist, then the default path is 
\DATADIRECTORY\IBM_TECHNICAL_SUPPORT and the default path is CONSOLE.LOG.

The following represents the debugging I have in my notes.ini currently to 
assist in troubleshooting my 6.5.4 errors - you can see that I have turned 
off several and enabled some others:

DEBUG_THREADID=1 
DEBUG_DBSERVER=1 
DEBUG_SHOW_TIMEOUT=1 
DEBUG_CAPTURE_TIMEOUT=10
DEBUG_NTI_DRVR=0 
DEBUG_TCP_ALL=0 
DEBUG_TCP_ERRORS=0 
DEBUG_OUTFILE=DEBUG.TXT

A side note on analysis, with options such as DEBUG_THREADID, you will be 
able to see a Process ID number along with Thread ID's. You can then, by 
possibly having agent print statements, be able to associate the 
Process/Thread ID with a particular agent, task, or user. This can 
potentially go a long way in helping to determine the cause of an issue.

Chris Whisonant
Comporium
Senior Mid-Range Systems Administrator
IBM eServer Certified Systems Expert - iSeries Technical Solutions V5R2
IBM Certified System Administrator - Lotus Notes and Domino 6/6.5
803.326.7270 | 803.326.6142 fax
http://cwhisonant.blogspot.com/

domino400-bounces+chris.whisonant=comporium.com@xxxxxxxxxxxx wrote on 
07/29/2005 11:52:04 AM:

> How does one turn on TCP logging?
> 
> Yes, it does occur after some period of inactivity.
> 
> Rob Berendt
> -- 
> Group Dekko Services, LLC
> Dept 01.073
> PO Box 2000
> Dock 108
> 6928N 400E
> Kendallville, IN 46755
> http://www.dekko.com
> 
> 
> 
> 
> 
> Chris Whisonant <Chris.Whisonant@xxxxxxxxxxxxx> 
> Sent by: domino400-bounces+rob=dekko.com@xxxxxxxxxxxx
> 07/29/2005 10:37 AM
> Please respond to
> Lotus Domino on the iSeries / AS400 <domino400@xxxxxxxxxxxx>
> 
> 
> To
> Lotus Domino on the iSeries / AS400 <domino400@xxxxxxxxxxxx>
> cc
> 
> Subject
> Re: Web site hangs
> 
> 
> 
> 
> 
> 
> You may want to turn on some TCP logging for the console. We were having 

> some poor client performance and when i turned on some logging for the 
> client I could see that the updating of unread marks was really killing 
us 
> 
> (1.5-2 minute db load - lol!) It could be helpful to see what was taking 

> so long. We could see that the particular unread mark thing was taking 
> 1500 ms +. Turned off unread marks on db and it "fixed" the issue.
> 
> Is unread marks turned on in your app? When the agent hangs is it after 
a 
> period of inactivity? Just some thoughts...
> 
> Chris Whisonant
> Comporium
> Senior Mid-Range Systems Administrator
> IBM eServer Certified Systems Expert - iSeries Technical Solutions V5R2
> IBM Certified System Administrator - Lotus Notes and Domino 6/6.5
> 803.326.7270 | 803.326.6142 fax
> http://cwhisonant.blogspot.com/
> 
> domino400-bounces+chris.whisonant=comporium.com@xxxxxxxxxxxx wrote on 
> 07/29/2005 11:24:49 AM:
> 
> > One of our Notes developers called me and said that sometimes an 
> > application on our domino based web site (in the dmz) hangs.  He 
checked 
> 
> 
> > the agent's log and it may experience up to a 5 minute delay when 
trying 
> 
> 
> > to update a file on a different domino server, in a different domain, 
in 
> 
> 
> > our intranet.
> > 
> > Don't know if it's a coincidence but we notice on QSYSOPR (on the dmz 
> > machine) the following message:
> > TCP2617-TCP/IP connection to remote system xxx.xxx.xxx.xxx closed, 
> reason 
> > code 2.
> > where xxx.xxx.xxx.xxx is the ip address of the domino server on the 
> > intranet machine.
> > Now, I know how to change the tcp attributes to turn this message off, 

> but 
> > I was wondering if this may be related to the hang.
> > 
> > I didn't see anything else in either QSYSOPR, nor the log.nsf, on 
either 
> 
> 
> > machine that caught my eye.
> > 
> > Rob Berendt
> > -- 
> > Group Dekko Services, LLC
> > Dept 01.073
> > PO Box 2000
> > Dock 108
> > 6928N 400E
> > Kendallville, IN 46755
> > http://www.dekko.com
> > 
> > _______________________________________________
> > This is the Lotus Domino on the iSeries / AS400 (Domino400) mailing 
list
> > To post a message email: Domino400@xxxxxxxxxxxx
> > To subscribe, unsubscribe, or change list options,
> > visit: http://lists.midrange.com/mailman/listinfo/domino400
> > or email: Domino400-request@xxxxxxxxxxxx
> > Before posting, please take a moment to review the archives
> > at http://archive.midrange.com/domino400.
> > 
> _______________________________________________
> This is the Lotus Domino on the iSeries / AS400 (Domino400) mailing list
> To post a message email: Domino400@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/domino400
> or email: Domino400-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/domino400.
> 
> 
> _______________________________________________
> This is the Lotus Domino on the iSeries / AS400 (Domino400) mailing list
> To post a message email: Domino400@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/domino400
> or email: Domino400-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/domino400.
>