Sentinel for orphaned jobs and hung devices

["James H H Lampert" <jamesl@xxxxxxxxxxx>]

In case anybody's interested (a remote possibility, no 
doubt):

Here's how I ended up dealing with hung devices and jobs 
on that public application: A sentinel system, mainly in 
CL. It goes in back doors, goes out windows (but not 
WinDoze), climbs up chimneys, and slides down waste line 
vent stacks, but it gets the job done.

The heart of the sentinel is the "sweep" program:

First, get a list of the devices in the series that's 
dedicated to the application, using a DSPOBJD into an 
outfile.

Then go through the records of the outfile, doing a 
RTVCFGSTS on each device. If the status is anything less 
than 60 (60=active), vary it off, then backon. That clears 
the ones hung in "signon display." But what about the ones 
hung in the application?

I added code to the application, that changes the *DEVD's 
object text to the SAA timestamp of its most recent 
legitimate I/O operation. Returning to the "sweep" 
program, if the status is 60 or higher, it gets the 
timestamp from the *DEVD's object text, and passes it to a 
simple RPG program that checks it against the current 
time: if it's more than 5 minutes old, it returns "KILL"; 
otherwise, it returns "KEEP". If the sweep program gets a 
"KILL" on an active device, it then does an ENDJOB on it 
(no need to worry about duplicate job names, since the 
account is rigged to never produce a spooled joblog), and 
leaves it for the next sweep to reset the device.

Then, an outer "sentinel" program, in a batch job, runs 
the "sweep" program, followed by a DLYJOB of 10 minutes 
(600 seconds), then repeats the process, ad infinitum. 
Thus, even an intentional denial-of-service attack on the 
application (and it appears that deliberate abuse is the 
only way to actually leave large numbers of devices in a 
"hung" state) would not cause any device to remain "hung" 
for more than 20 minutes, tops.

--
JHHL