java, multithreading, fork() - performance problem on Solaris

Thread view:

bugbear - 25 Mar 2008 16:30 GMT

I'm using tomcat on Solaris with around 100 threads.
This is perfectly normal and mundane, and works well.

However, I have now noticed that when the JSP's
call system() (either via runtime.exec or new processBuilder)
the whole thing slows down a lot, and the kernel
usage goes crazy. Idle time goes UP (which is odd).
JSP response tanks.

No, I can't stop shelling out from tomcat :-(

I have traced this problem to fork(), to whit:

The fork() and fork1() functions suspend
all threads in the process before proceeding.

(ref:
http://docs.sun.com/app/docs/doc/816-0212/6m6nd4n9e?a=view
)

They're all started up again after the fork(), of course, and
I have confirmed this diagnosis by crawling over an ENORMOUS
truss output log.

Now, with 100 tomcat threads, this is a bit of an overhead (*)

It appears that multithreaded Java and fork()
on Solaris have a rather deep incompatibility.

Has any body else seen/experienced this, and
has anybody found a solution/workround?

BugBear

(*) British understatement.

Reply to this Message

Tim Bradshaw - 26 Mar 2008 09:21 GMT

> Has any body else seen/experienced this, and
> has anybody found a solution/workround?

It might be that posix_spawn has different behaviour, though this
seems unlikely to me. Otherwise the obvious thing would be to have a
separate, single-threaded, process which does the forks to which your
huge JVM can talk using some IPC mechanism.

Reply to this Message

Casper H.S. Dik - 26 Mar 2008 11:18 GMT

>> Has any body else seen/experienced this, and
>> has anybody found a solution/workround?

>It might be that posix_spawn has different behaviour, though this
>seems unlikely to me. Otherwise the obvious thing would be to have a
>separate, single-threaded, process which does the forks to which your
>huge JVM can talk using some IPC mechanism.

System and posix_spawn both use vfork(); vfork() also needs to
suspend all threads, pretty much by definition.

Note that fork/fork1 are expensive in requiring all page
table entries for the process to be marked "copy on write".

vfork() is generally cheap because it only needs to suspend the
parent (but that is quite different when the parent has 100s
of threads). (While vfork() is documented as not being thread
safe, the restrictions on its use are known and fixed so we
do know how to safely use it in the implementation of system(),
popen() and posix_spawn)

The fork() process creation model and the POSIX threads in a single
address space model don't really work well together and the
combination should be avoided.

Casper

Signature

Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Reply to this Message

bugbear - 26 Mar 2008 13:15 GMT

>> Has any body else seen/experienced this, and
>> has anybody found a solution/workround?
[quoted text clipped - 3 lines]
> separate, single-threaded, process which does the forks to which your
> huge JVM can talk using some IPC mechanism.

Yes, thank you.

That solution is quite natural, but would complicate
out software architecture (and hence deployment).

I was hoping to avoid such a solution if possible...
(which it may not be :-( )

It appears that (in theory) the problem
is independant of Java; any strongly multi threaded
code would have "issues" with shell out.

It's just that Java tends to *be* strongly
multi-threaded.

BugBear

Reply to this Message

Arne Vajhøj - 28 Mar 2008 20:25 GMT

> I'm using tomcat on Solaris with around 100 threads.
> This is perfectly normal and mundane, and works well.
[quoted text clipped - 6 lines]
>
> No, I can't stop shelling out from tomcat :-(

[other have commented on the fork part]

You may need to run external commands, but could
you instead of starting new processes have a
pool of permanent processes you just communicated
with ?

Arne

Reply to this Message

java, multithreading, fork() - performance problem on Solaris

Free Magazines