Re: Quantify processing wasted by lots of little jobs

[Simon Coulter <shc@xxxxxxxxxxxxxxxxx>]

On 02/11/2007, at 8:09 AM, Wilt, Charles wrote:

> Well all the jobs are submitted to a single threaded queue.... 
> (single threaded due to a design
> limitation ).
>
> Right now, there's 1800+ jobs waiting in the queue to run.  CPU  
> usage at 100%.
>
> Seems like the CPU time spent creating the separate jobs would be  
> better used running the process needed.

Yes, it would be. Although the overhead is less noticeable on newer  
hardware. Not sure if that is because Rochester have made the job  
start more efficient or simply because new hardware is so much faster.

Performance Tools will show you the job overhead. In fact this very  
scenario was used in performance classes (e.g., interactive job  
printed a picking slip, process changed to submit print, performance  
sucked, changed to populate data queue instead of submit, performance  
better).

> Ideally, I'd like to have the process using data queues and  
> multiple NEP's to handle the processing.
> But it'd be a bigger job to allow multiple process to run at the  
> same time if it's even possible (the
> limitation is not an iSeries application design issue, an AIX box  
> is involved).
>
> I seem to remember an article with a example program that used data  
> queues and automatically increased
> or decreased the number of NEP jobs running.  Anybody got a link to  
> that?

I don't have a link but I know I've discussed this in previous  
appends. Check the archives. I've written code to do this.  
Essentially you start a minimum number of NEPs to handle the data  
queue. Each NEP tracks statistics of how many entries it is  
processing over a given time. As each NEP pulls an entry off the  
queue it checks the queue attributes to see how many entries are  
left. If this exceeds a certain threshold based on the processing  
statistics the NEP submits another copy of itself. Each NEP must also  
time-out when waiting for an new entry and check how many copies are  
active. If too many (again based on a certain threshold) it ends  
itself. Nett result is a wave of NEPs increasing as work load  
increases and reducing as there is less to do.

Regards,
Simon Coulter.
--------------------------------------------------------------------
   FlyByNight Software         OS/400, i5/OS Technical Specialists

   http://www.flybynight.com.au/
   Phone: +61 2 6657 8251   Mobile: +61 0411 091 400        /"\
   Fax:   +61 2 6657 8251                                   \ /
                                                             X
                 ASCII Ribbon campaign against HTML E-Mail  / \
--------------------------------------------------------------------