James,
Probably related to a firewall.
We have a similar situation, not with a Tomcat job, but an interface job.
Previously, when a process would run, once a day, it would open the ports, do the process, close the ports.
A remote Cisco firewall was upgraded to new hardware/software.
Now, when the remote side sends the command to close the ports, the port is remaining opening on the Power I.
The port was being closed on the remote side, but the command to close was not being passed thru the new firewall.
Next time the process needs to run it fails, because the port was still open.
Using Halcyon, I created an object monitor, that looked for the open port for the specific job, and closed if found.
HEM0010R Halcyon System Event Manager PENCOR05
Display Object Criteria 11/30/20 15:03:44
Rule group . . . . . . . . SA_UPLOAD
Rule number . . . . . . . . 10 SA_UPLOAD ports remaining open
Rule type . . . . . . . . . *RCDCNT
File name . . . . . . . . . NS_JOB
Library . . . . . . . . . QSYS2
ASP group . . . . . . . . *SYSBAS
Check member . . . . . . . *FIRST
SQL select clause . . . . . SELECT REMOTE_ADDRESS , JOB_NAME , LOCAL_PORT , REM
OTE_PORT FROM QSYS2/NS_JOB WHERE REMOTE_ADDRESS = '192.168.x.x' AND JOB_NAME LIK
E '%/SA_UPLOAD' ORDER BY REMOTE_ADDRESS , JOB_NAME FETCH FIRST 1 ROWS ONLY
Alert when records . . . . *GT *EQ, *NE, *LE, *GE, *LT, *GT
Value . . . . . . . . . . . 0 0 - 999999
SLA statistic . . . . . . . *NO
Open alert action . . . . . *DFT
F3=Exit F12=Cancel F21=Command line
HEM_MNTRUL Halcyon System Event Manager PENCOR05
Display Object Rule 11/30/20 15:00:09
Object group . . . . . . SA_UPLOAD (2 of 2)
Description . . . . . . . SA_UPLOAD ports remaining open
Type options, press Enter.
5=Display
pt Rule type Selection criteria Cmp Cnt
*RCDCNT SELECT REMOTE_ADDRESS , JOB_NAME , LOCAL_PORT , REMOTE_P*GT 0
______________________________ Actions ___________________________________
Action Delay
CONSOLE SYSTEM(*LOCAL) SUBCONS(QSECOFR) 0
COMMAND CMD(RUNSQL REQUEST('SELECT LOCAL_PORT , REMOTE_PORT FROM QSYS2 0
COMMAND CMD(SNDTXTMSG MSG('&SYSNAME &JOBNAME &MSGTXT &MSGHLP') TOMSGDE 0
COMMAND CMD(QSYS/ENDTCPCNN PROTOCOL(*TCP) LCLINTNETA('10.5.x.x') LCLP 0
Paul
-----Original Message-----
From: MIDRANGE-L <midrange-l-bounces@xxxxxxxxxxxxxxxxxx> On Behalf Of James H. H. Lampert
Sent: Monday, November 30, 2020 1:25 PM
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxxxxxxxx>
Subject: Weird problem with a job not ending when told to
We've got this weird situation on a customer box.
We have a Tomcat server. And we have an ILE CL program to shut it down (in this case, for an automated restart).
The program first attempts to do an orderly shutdown of the Tomcat server, by submitting a batch job (to QINTER) that QShell's the SHUTDOWN.SH script.
It then repeatedly calls an ILE RPG procedure (once per second) to check (using QUSLJOB) whether the CATALINA job is still there (if it is, it returns the fully qualified jobname; if not, it returns blank). If a full minute has passed without the CATALINA job disappearing, it then does an ENDJOB on the fully-qualifed jobname, with OPTION(*IMMED).
This works everywhere else. But on this one customer box, it occasionally fails: the Tomcat server gets as far as closing the port, but it doesn't end normally, and *IT DOESN'T GET ABENDED BY THE ENDJOB*
At this point, when the restart program restarts Tomcat, there are two CATALINA jobs, and two underlying QP0ZSPWT jobs, but since the older one got as far as closing the HTTPS port, the newer one is able to open it without a problem. But then, the very next automated restart, the shutdown finally shuts down the "zombie" Tomcat server, leaving the port tied up by the live one, so that the restart won't work.
I know of a couple ways to do an end-run around the problem, but I'd still like to know what could be making it fail.
--
JHHL
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit:
https://smex-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2flists.midrange.com%2fmailman%2flistinfo%2fmidrange%2dl&umid=647b411c-a2ab-4794-b426-9f7991248f13&auth=438b0784514c1757bd202125ca4db8b0abdb021e-e1260938a0551bd5a2145d23e4b14c5339efbb38
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://smex-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2farchive.midrange.com%2fmidrange%2dl&umid=647b411c-a2ab-4794-b426-9f7991248f13&auth=438b0784514c1757bd202125ca4db8b0abdb021e-238ed393b583c28946721b25a7c25491bd9b7c24.
Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.
Help support midrange.com by shopping at amazon.com with our affiliate link:
https://smex-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2famazon.midrange.com&umid=647b411c-a2ab-4794-b426-9f7991248f13&auth=438b0784514c1757bd202125ca4db8b0abdb021e-487826b8d733dd75bc4c0c88334e390ad2f735f0
midrange.com
