Re: JVM CPU queueing

some additional remarks to the discussion:
- load balancing between the processors is the task of the operating system. Just an example for as400 without any Java involved:
We have 4 processors and some jobs (processes) are running. At every synchronous I/O the process is suspended and afterwards it will get another timeslice, when available. The operating system chooses one of the idle cpus and it goes on. If the same cpu was idle all the time, best would be to give the process the same cpu. Now it could happen, that the workload for the 4 cpu is getting unbalanced and if no process terminates and no process is coming in (nightly batch, or NEPs) the OS needs the capability to "move" a process from one CPU to another to balance the load. This will happen rather seldom, when cpu activity is low and happen rather frequently, when workload is near 100%.
Same scenario with "native" threads, if they are implemented well by the OS.

Now lets have a look at Java (or any other multithreaded application): If a long running "batch" doesn't use threads, it might run on diffrent cpus, but it won't use more than 1 at a time, so it is limited to only a part of the power of the system. Using multiple threads, the workload that could be done increases. The batch is running faster, or in a multiuser environment, more transactions are done in a certain time, if every user gets his own thread(this would happen without any programming in the application, the WebServer will handle this! BTW: using EJBs same thing will happen for a well designed "batch" without any Thread Handling in the application). In the times (long ago) Java was emulating Threads ("green" Threads), the OS didn't recognize, that the application was using diffrent threads and no balancing did happen - this changed with Java 1.2 (AFAIK).

As long as the CPU usage is << 100%, there are idle ressources, that could be used for cpu intensive workload, so it's no surprise, that the maximum is reached by increasing the number of threads, until cpu usage is near 100%. And it's no surprise too, that cpu usage is not balanced, when cpu usage is low. It might look like a JVM is running on one CPU only, if this CPU has enough power to do all related work and concurrent (non Java) workload is low too. Near the 100% the overhead is rather high compared to low cpu usage and you will come in some risc, that the overhead is eating up all cpu, so you would not go to close to this. One exception might be, if you have a very clear estimate what will happen next hour (we've done this for the load process for a datawarehouse , we knew: next two hours there will be load time and nothing else and to get maximal throughput we went very close to 100%).

For the OP it would be very important to make some load tests, to have an impression if the system is well balanced (CPU, I/O, Network) and scaled well.

Dieter Bender