 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / Virtual Machine / January 2005Java process hangs at 99.9% cpu, cannot be killed
Hi all, The problem is on a dual CPU server with Fedora Core 2 (kernel 2.6.5) with all the updates installed on it. Tomcat 5.0.28 and IBM JDK 1.4.2. We have two applications, one runs on the tomcat and the other one is standalone, launched from the shell. Tomcat application is always running. The other one, we seldomly start it to do a fixed job. We frequently (once a day usually) face with a situation where our java process consumes 99.9% of cpu and cannot be killed. (both on tomcat or the standalone app.) Or sometimes it doesnt consume that much cpu and can be killed with -9 signal, but becomes zombie this time. We suspected that there might be a problem with the SMP kernel and switched to a single cpu kernel ( I can verify that on /proc/cpuinfo). This time the problem is much weird. Again once a day, the ssh connection to the server stops responding. We can ping the machine, can establish a connection (to apache, tomcat, etc) but then there is no more data transfer. And we have to restart the server then. I have checked many things like open files, memory usage, server load etc. They all seem normal. I tried Sun JDK, it gets even worse. If I use Sun JDK (1.4.2_06) I face these problems twice a day or even more often. Also tried Blackdown, installed it and then run java --version to verify it's installed correctly, then it hanged up and consumed 99% of cpu, couldnt kill it, restarted the server :( I suspect there must be a problem with the JVM, with the harware or the OS... Does anybody have any idea of what is going around, or what should I check ? Thanks in advance, Antonis If you kill -3 a java program it'll dump the java stack trace on stderr. that'll help u undesrtand the state of the process. > Hi all, > > The problem is on a dual CPU server with Fedora Core 2 (kernel 2.6.5) > with all the updates installed on it. Tomcat 5.0.28 and IBM JDK 1.4.2. > > We have two applications, one runs on the tomcat and the other one is > If you kill -3 a java program it'll dump the java stack trace on stderr. > that'll help u undesrtand the state of the process. kill -3 doesn't work, i tried it many times. For tomcat, I see a text like that "please wait, dumping thread stacktrace..." and then nothing more. For the standalone application, i cant even see that text. hmmm...thats interesting...right now I have no way to check it.... but can u find out if -3 indeed is the signal to dump java stack? also, i guess the output goes to stderr, not stdoutput, did u check tomcat stderr log? Yes, try it on a standalone java process first..verify the sgnal to send, the log(err or out) then try it on tomcat... .. >> If you kill -3 a java program it'll dump the java stack trace on > stderr. [quoted text clipped - 5 lines] > > For the standalone application, i cant even see that text. > hmmm...thats interesting...right now I have no way to check it.... > but can u find out if -3 indeed is the signal to dump java stack? yes, -3 is the right signal. I tried it on normal java processes and it dumps. > also, i guess the output goes to stderr, not stdoutput, did u check tomcat > stderr log? i checked both stdout and stderr, nothing is printed. > The problem is on a dual CPU server with Fedora Core 2 (kernel 2.6.5) > with all the updates installed on it. Tomcat 5.0.28 and IBM JDK 1.4.2. <snip> > I suspect there must be a problem with the JVM, with the harware or the > OS... What makes you think the problem is not in your app? You tried everything, everything failed, and the only static factor was your app. What about a thread with *max-priority* that finds itself in a tight loop? > > I suspect there must be a problem with the JVM, with the harware or the > > OS... > > What makes you think the problem is not in your app? You tried everything, > everything failed, and the only static factor was your app. Because when my app. is not running, when i try to stop tomcat it sometimes hangs. Also blackdown jvm hangs even when I just run java --version. > What about a thread with *max-priority* that finds itself in a tight loop? I dont think that's the case, but I will double check again.. By the way, do you think that a therad with a while (true) {} cannot be killed even with kill -9 ? > > What about a thread with *max-priority* that finds itself in a tight > loop? [quoted text clipped - 4 lines] > while (true) {} > cannot be killed even with kill -9 ? WARNING: 'dangerous' code (under WinXP that is...) Thread t = new Thread() { public void run() { while (true){} } }; t.setPriority(Thread.MAX_PRIORITY); Runtime.getRuntime().addShutdownHook(t); ~~~~~~~~~~~ The thread-priority is respected when running the shutdown-hooks. This might be the reason why (as you posted in the other sub-thread) the initial shutdown-sequence is started by Tomcat, but something (another shutdownhook?) takes over. Note: I'm not experienced at all in this subject. I had to reboot WinXP, because I couldn't get the process to stop in any way. (frozen GUI) > Also blackdown jvm hangs even when I just run java > --version. So either the Blackdown JVM is seriously broken (for the context of your machine/OS) or the machine itself is broken. If that's the case then you don't need any /other/ explanation for why your application fails. Since your application fails even worse when you are running the Sun JVM, it seems likely not to be a problem with Blackdown. Hence, it seems to be a machine problem. Try a different box with the same OS. -- chris Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.