HotSpot crashed on Linux

Thread view:

nicolas.raoul@gmail.com - 17 Jul 2007 11:01 GMT

The HotSpot JVM was running Tomcat, which hosted our web application
in the production environment.

It had been restarted 6 days ago and had been running fine since. And
the same JVM+application usually work fine, we have used them for
months, restarting from time to time, and never had this problem.

In the JVM error file, I notice about 5000 threads, could this be a
problem ? Is the HotSpot JVM unable to handle many threads ? Should I
change the way my application uses threads ?

Could it be caused by my web application ?
I could not find any information on how to prevent this kind of
crashes.
What can I do so that it does not happen again ?

Below is the generated hs_err_pid*.log error file, in which I replaced
two long boring parts with "..."

Any idea/hint/advice appreciated ! :-)
Nicolas RAOUL.

-----------------------------------------------------------------

#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x064c5d46, pid=30568, tid=1797516208
#
# Java VM: Java HotSpot(TM) Server VM (1.6.0_01-b06 mixed mode)
# Problematic frame:
# V [libjvm.so+0x4c5d46]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x080d0000): JavaThread "Timer-133" daemon
[_thread_blocked_trans, id=31497]

siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=0x00000000

Registers:
EAX=0xb7f94000, EBX=0x080d0000, ECX=0x00000000, EDX=0x00000000
ESP=0x6b23ddc8, EBP=0x6b23de60, ESI=0x00000ffc, EDI=0x090180a4
EIP=0x064c5d46, CR2=0xb7f94000, EFLAGS=0x00010246

Top of Stack: (sp=0x6b23ddc8)
0x6b23ddc8: 0000000d 6b23dcd8 fffffffd 00000000
0x6b23ddd8: 090180f4 01000006 6b23de70 061876f0
0x6b23dde8: 080d0000 00000000 75b88818 75b88818
0x6b23ddf8: 080d0e20 00000002 6b23de30 062ddaba
0x6b23de08: 00000000 00000000 fffffff0 0023de30
0x6b23de18: 080d0000 34000010 0000002a b7f676e8
0x6b23de28: 6b23de28 6b23de28 080d0000 080d0d30
0x6b23de38: 00000000 00000003 00000000 00000029

Instructions: (pc=0x064c5d46)
0x064c5d36: 8b 35 b0 af 5a 06 c1 e9 03 a1 ac af 5a 06 21 f1
0x064c5d46: c7 04 01 01 00 00 00 e9 be fc ff ff 83 ec 0c 8b

Stack: [0x6b1ee000,0x6b23f000), sp=0x6b23ddc8, free space=319k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V [libjvm.so+0x4c5d46]
V [libjvm.so+0x4c35ef]
V [libjvm.so+0x30785a]
J java.lang.Object.wait(J)V

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J java.lang.Object.wait(J)V
J java.util.TimerThread.mainLoop()V
j java.util.TimerThread.run()V+1
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x2a37ec00 JavaThread "Timer-4500" daemon [_thread_blocked_trans,
id=24820]
... (4563 similar lines with "Timer-XYZ")
0x703ac800 JavaThread "Timer-3" daemon [_thread_blocked_trans,
id=30640]
0x7038c800 JavaThread "TP-Monitor" daemon [_thread_blocked,
id=30616]
0x6fce0c00 JavaThread "TP-Processor4" daemon [_thread_in_native,
id=30615]
0x70397c00 JavaThread "TP-Processor3" daemon [_thread_blocked,
id=30614]
0x70397800 JavaThread "TP-Processor2" daemon [_thread_blocked,
id=30613]
0x703c0c00 JavaThread "TP-Processor1" daemon [_thread_blocked,
id=30612]
0x703c5000 JavaThread "http-8080-Monitor" [_thread_blocked,
id=30611]
0x6fffe800 JavaThread
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon
[_thread_blocked_trans, id=30585]
0x6ffa1000 JavaThread "Thread-2" [_thread_blocked_trans, id=30582]
0x703ae400 JavaThread "Timer-0" daemon [_thread_blocked_trans,
id=30581]
0x08133400 JavaThread "Low Memory Detector" daemon [_thread_blocked,
id=30579]
0x08131800 JavaThread "CompilerThread1" daemon [_thread_blocked,
id=30578]
0x08130000 JavaThread "CompilerThread0" daemon [_thread_blocked,
id=30577]
0x0812ec00 JavaThread "Signal Dispatcher" daemon [_thread_blocked,
id=30576]
0x0811d400 JavaThread "Finalizer" daemon [_thread_blocked, id=30575]
0x0811cc00 JavaThread "Reference Handler" daemon [_thread_blocked,
id=30574]
0x08058800 JavaThread "main" [_thread_in_native, id=30570]

Other Threads:
0x0811a000 VMThread [id=30573]
0x08134c00 WatcherThread [id=30580]

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x08056aa0/0x08056ac8] Safepoint_lock - owner thread: 0x0811a000
[0x08056b20/0x08056b48] Threads_lock - owner thread: 0x0811a000
[0x08056fe0/0x08056ff8] Heap_lock - owner thread: 0x087a8400

Heap
PSYoungGen total 116288K, used 116272K [0xadcb0000, 0xb4e70000,
0xb4e70000)
eden space 116096K, 100% used [0xadcb0000,0xb4e10000,0xb4e10000)
from space 192K, 91% used [0xb4e10000,0xb4e3c060,0xb4e40000)
to space 192K, 0% used [0xb4e40000,0xb4e40000,0xb4e70000)
PSOldGen total 84416K, used 67409K [0x74e70000, 0x7a0e0000,
0xadcb0000)
object space 84416K, 79% used [0x74e70000,0x790444f0,0x7a0e0000)
PSPermGen total 18816K, used 18597K [0x70e70000, 0x720d0000,
0x74e70000)
object space 18816K, 98% used [0x70e70000,0x72099750,0x720d0000)

Dynamic libraries:
003bb000-003d0000 r-xp 00000000 fd:00 556129 /lib/ld-2.3.4.so
... (9471 similar lines)
ffffe000-fffff000 ---p 00000000 00:00 0

VM Arguments:
jvm_args: -DTACTDIR=/usr/local/tact -DTACTDATADIR=/usr/local/
tactdatadir -DTACTURL=http://172.16.12.142:8080/tact -Xms256m -
Xmx1024m -
Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -
Djava.util.logging.config.file=/usr/local/tomcat/conf/
logging.properties -Djava.endorsed.dirs=/usr/local/tomcat/common/
endorsed -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/
tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/local/java
PATH=/usr/local/tactshells:/opt/firefox:/usr/local/java/bin:/usr/
kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/
bin:/usr/sbin:/usr/bin:/usr/local/tools:/usr/local/tactshells:/usr/
X11R6/bin:/home/tact/bin:.
LD_LIBRARY_PATH=/usr/local/jre1.6.0_01/lib/i386/server:/usr/local/
jre1.6.0_01/lib/i386:/usr/local/jre1.6.0_01/../lib/i386
SHELL=/bin/bash
DISPLAY=:0.0

Signal Handlers:
SIGSEGV: [libjvm.so+0x51d3a0], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGBUS: [libjvm.so+0x51d3a0], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGFPE: [libjvm.so+0x43d430], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x43d430], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGILL: [libjvm.so+0x43d430], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x43f440], sa_mask[0]=0x00000000,
sa_flags=0x10000004
SIGHUP: [libjvm.so+0x43ee60], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x43ee60], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGTERM: [libjvm.so+0x43ee60], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004
SIGUSR2: [libjvm.so+0x43f440], sa_mask[0]=0x00000000,
sa_flags=0x10000004

--------------- S Y S T E M ---------------

OS:Red Hat Enterprise Linux WS release 4 (Nahant Update 4)

uname:Linux 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE 0k, NPROC 32757, NOFILE 1024, AS infinity
load average:699.24 155.09 53.29

CPU:total 2 family 6, cmov, cx8, fxsr, mmx, sse, sse2

Memory: 4k page, physical 2074272k(473144k free), swap
2097144k(2097016k free)

vm_info: Java HotSpot(TM) Server VM (1.6.0_01-b06) for linux-x86,
built on Mar 14 2007 00:47:53 by "java_re" with gcc 3.2.1-7a (J2SE
release)

Reply to this Message

Roedy Green - 17 Jul 2007 20:24 GMT

>In the JVM error file, I notice about 5000 threads,

It may not be so bad now, but at one point you needed about a meg per
thread just to hold the stack. Have a look at the new
java.util.concurrent package. There are goodies there for sharing
threads. See http://mindprod.com/jgloss/queue.html

Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Reply to this Message

Juha Laiho - 21 Jul 2007 16:52 GMT

nicolas.raoul@gmail.com said:

>The HotSpot JVM was running Tomcat, which hosted our web application
>in the production environment.

...

>In the JVM error file, I notice about 5000 threads, could this be a
>problem ? Is the HotSpot JVM unable to handle many threads ? Should I
>change the way my application uses threads ?
>
>Could it be caused by my web application ?

Hmm.. the amount of threads could be a concern, and as such it could be
caused by your applications.

>I could not find any information on how to prevent this kind of
>crashes.
>What can I do so that it does not happen again ?

One thing you might do is to file the error report to Sun, as hinted in
the crash log.

>-----------------------------------------------------------------
># An unexpected error has been detected by Java Runtime Environment:
[quoted text clipped - 4 lines]
># Problematic frame:
># V [libjvm.so+0x4c5d46]

Check the release notes of 1.6.0_02 JRE for the SIGSEGV details; it
could even be fixed already.

># If you would like to submit a bug report, please visit:
># http://java.sun.com/webapps/bugreport/crash.jsp
[quoted text clipped - 3 lines]
>Current thread (0x080d0000): JavaThread "Timer-133" daemon
>[_thread_blocked_trans, id=31497]

So, the SEGV happened within one of the Timer threads.

>--------------- P R O C E S S ---------------
>
>Java Threads: ( => current thread )
> 0x2a37ec00 JavaThread "Timer-4500" daemon [_thread_blocked_trans,
>id=24820]
>... (4563 similar lines with "Timer-XYZ")

And there's quite a number of such threads (and to me at least it sounds
strange to have that many timer threads. Also, I don't see any Timer-XXX
thread on my Tomcat, so it is likely that the threads are created by your
code (or alternatively some library unique to your set-up).

Signature

Wolf a.k.a. Juha Laiho Espoo, Finland
(GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

Reply to this Message

nicolas.raoul@gmail.com - 23 Jul 2007 13:24 GMT

Hi,
Thanks for your answers :-)

> One thing you might do is to file the error report to Sun, as hinted in the crash log.

I did that immediately :-)
I have got no feedback from Sun, though... I hope it means that they
have fixed it already.

> Check the release notes of 1.6.0_02 JRE for the SIGSEGV details; it
> could even be fixed already.

The release notes for update 2 (on http://java.sun.com/javase/6/webnotes/ReleaseNotes.html)
do not mention any such fix :-(

> there's quite a number of such threads
> (and to me at least it sounds
> strange to have that many timer threads).

I will definitely try to find out where all of this threads come from,
and fix it because I am quite sure I don't need that many. I mean, I
launch maybe 10 threads per minute, but they spend only a few seconds
(5 minutes maximum) in method run(), and I don't keep a reference to
them so they should be nullified and garbage collected right ? Or do I
need to explicitly stop them ?

Let's say method launchThread() creates and starts a thread T, then
exits without keeping any reference to thread T. When method run() of
thread T is over, is there any thing left to do ? Should I explicitly
destroy T or is it done automatically ?

Another thread creator suspect is a feature we have to regularly check
configuration files updates, I will try and disable this feature,
could be a bug with that, never know.

Thanks a lot !
Nicolas RAOUL
http://nrw.free.fr

Reply to this Message

David Gourley - 23 Jul 2007 20:40 GMT

Every Timer object you create in your code (implicitly) creates a
thread. I don't think these threads go away till the timer objects get
garbage collected. It's much better to have a single timer object with
multiple enqueued TimerTasks than a huge number of Timer objects....

Directly related to the memory overhead (huge number of threads) there
is a significant performance overhead of huge numbers of Timer objects
as thread creation can be (comparatively) slow compared to most object
creation...

Dave

Reply to this Message

Roedy Green - 24 Jul 2007 08:41 GMT

>Every Timer object you create in your code (implicitly) creates a
>thread. I don't think these threads go away till the timer objects get
>garbage collected. It's much better to have a single timer object with
>multiple enqueued TimerTasks than a huge number of Timer objects...

On the other paw, only one nugget of work can run at a time with a
single Timer.

Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Reply to this Message

David Gourley - 24 Jul 2007 21:16 GMT

>>Every Timer object you create in your code (implicitly) creates a
>>thread. I don't think these threads go away till the timer objects get
[quoted text clipped - 3 lines]
> On the other paw, only one nugget of work can run at a time with a
> single Timer.

Agreed. We've tried consciously to limit the work performed in Timer
threads for this reason, but I agree that if you don't you may need to
allow multiple Timer objects.

On the other hand, not controlling numbers of Timer objects is a recipe
for disaster.....

Dave

Reply to this Message

Juha Laiho - 25 Jul 2007 17:47 GMT

Roedy Green <see_website@mindprod.com.invalid> said:

>>Every Timer object you create in your code (implicitly) creates a
>>thread. I don't think these threads go away till the timer objects get
>>garbage collected. It's much better to have a single timer object with
>>multiple enqueued TimerTasks than a huge number of Timer objects...

>On the other paw, only one nugget of work can run at a time with a
>single Timer.

So, would a ThreadPool assigned as worker threads for a Timer be the
"correct solution" for this? This of course leaves the issue of
pool sizing, but I think in most cases it shouldn't be too hard to
find some ballpark figure; 10, 50, ... . Still much less than the
4500 or so that the OP started with.

Signature

Reply to this Message

David Gourley - 25 Jul 2007 20:30 GMT

> Roedy Green <see_website@mindprod.com.invalid> said:
>
[quoted text clipped - 6 lines]
> find some ballpark figure; 10, 50, ... . Still much less than the
> 4500 or so that the OP started with.

.. with the timer simply queueing work to one of the worker threads.
I've seen this (successfully) done on projects - means you only need one
timer object as all its thread does is invoke methods on the worker threads.

(I really, really hate the built in timer class - it has other features
(associated with TimerTask cancellation) that can lead the unwary to
OutOfMemory exceptions if they aren't careful. The 1.5 and on "fix"
feels like a nasty hack to me...

Dave

Reply to this Message

HotSpot crashed on Linux

Free Magazines