Web jobs and PRINTER files in RPG

[Mike Cunningham <mcunning@xxxxxxx>]

Sending this to Midrange and Web400 as I think it crosses over.

We have been using QPRINT and EXCEPT statements in web RPG apps we have been developing as a way to trace activity and as a way to report unexpected results in the application. We usually have the QPRINT file set for automatic open when the app starts. We have not had any trouble with this until just this week when we had heavier than normal web activity. (We are a college and this week was class scheduling for Spring). When 300 students were hitting the web server Monday at 8:30 the web response stopped almost completely. CPU use was only around 30%. Green screen still worked OK. There was very high faulting/paging numbers (web server pool had 9GB of ram assigned). System pool faulting was not about 1. When we did some digging we saw many of the web server CGI jobs in a lock-wait status on QPRINT. And the web server had started about 100 CGI handlers (we normally run with 3). After about 20-30 minutes like this it stopped and was fine the rest of the day. The scheduling web app runs in a named activation group and our web system is a mix of java/net.data/RPG CGI.  Tuesday morning at 8:30 it happened again, exactly same symptoms and after 20-30 minutes it was fine. Yesterday we changed each app that had a QPRINT file being opened and either removed it completely or changed it to user open and only open it if needed and this morning (Wed) the problem did not return and the number of CGI handlers is around 20 instead of 100.

I think what was going on was that while one CGI job was waiting on QPRINT the next hit had to spawn a new handler which also tried to get a lock on QPRINT so the next hit started a new handler which tried to get a lock, etc, etc, etc, and each new job just too resources away from the first job that was trying to get the lock. These apps also open lots of DB files and we never saw a lock wait on those files. Does opening a spool file require a lot of overhead that would take more time/resources to open as compared to a DB file?   Does opening a spool file require a lock on some shared object like the OUTQ that QPRINT goes to? I am trying to get an understating about what might have happened so we don't make the same mistake again.