Java Forum / General / January 2006
The program that would'nt Quit
Roedy Green - 26 Jan 2006 02:34 GMT I have I think discovered a strange bug in both JDK 1.5 and 1.6.
the program ends like this:
System.out.println( "done" ); // System.exit( 0 ); } // end main
Without that System.exit, the program keeps on running endlessly using about 94% of my CPU time after printing "done".
The odd thing is this is not a gui and it has but a single thread. I have no finalizers.
It works fine in Jet.
I don't know where to begin to track this down or how to reproduce it for Sun. The jar is 271K and the data files it works on are 14 MB.
Any ideas on what could cause such behaviour?
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 26 Jan 2006 02:57 GMT On Thu, 26 Jan 2006 02:34:42 GMT, Roedy Green <my_email_is_posted_on_my_website@munged.invalid> wrote, quoted or indirectly quoted someone who said :
>the program ends like this: > > System.out.println( "done" ); > // System.exit( 0 ); > } // end main there in the class containing main. I don't see anything out of the ordinary about it.
package com.mindprod.htmlmacros; import java.io.File; import java.io.FilenameFilter; import java.util.ArrayList;
import com.mindprod.filter.ExtensionListFilter;
/** * Feeds the entire Mindprod website to the Replacer macro mechanism refreshing * expansions of all macros. Expands HTML boilerplate refresher macros e.g. <!-- * macro Include dogs.txt --> <!-- generated --> <img * src="images/animals/dog.jpg" height="410" width="511" alt="dog"> <!-- * /generated by Include --> Allows arbitrary expander macro code. Futures: * Error messages give offset/line number, nearby text check for generated by * macro not just generated to remove old text. Can embed a " in a parm by * encoding it as " */ public class ReplaceMindprod extends Replacer {
/** * True if want debugging output, counts of files gathered from different * dirs. */ private static final boolean DEBUGGING = false;
/** * total count of files gathered */ private static int gatherCount;
/** * arrayList where we build list of files to process. */ static ArrayList<String> filesToProcess;
/** * Estimated files will process. */ public static final int ESTIMATED_TOTAL_FILES = 4500;
/** * Collect all files in the given dirName for processing. * * @param dirName * Name of fully qualified directory where to look for the html * files. * @param filter * filter to use to decide while files in that dir to process. */ static void gather ( String dirName, FilenameFilter filter ) { String[] filenames = new File( dirName ).list( filter ); if ( filenames == null ) { throw new IllegalArgumentException( "com.mindprod.htmlmacros.ReplaceMindprod: No such dir: " + dirName ); } gatherCount += filenames.length; if ( DEBUGGING ) { System.out.println( filenames.length + " files gathered from " + dirName ); } for ( String filename : filenames ) { filesToProcess.add( dirName + "/" + filename ); } }
/** * Get fully qualified name of all the files to process */ public static void gatherFiles () {
// add *.html files in mindprod
filesToProcess = new ArrayList<String>( ESTIMATED_TOTAL_FILES ); FilenameFilter f = new ExtensionListFilter( new String[] { "html" } );
gather( "e:/mindprod", f ); gather( "e:/mindprod/animalrights", f ); gather( "e:/mindprod/applets", f ); gather( "e:/mindprod/applications", f ); gather( "e:/mindprod/bgloss", f ); // bookcovers gather( "e:/mindprod/contact", f ); gather( "e:/mindprod/deepthoughts", f ); // dvdcovers gather( "e:/mindprod/environment", f ); gather( "e:/mindprod/esperanto", f ); gather( "e:/mindprod/ethics", f ); gather( "e:/mindprod/feedback", f ); gather( "e:/mindprod/ggloss", f ); gather( "e:/mindprod/humanrights", f ); // icon // images // include won't work since macros in wrong dir gather( "e:/mindprod/jgloss", f ); // jgloss/include // kjv, rarely changes, do manually when it needs it. gather( "e:/mindprod/livinglove", f ); gather( "e:/mindprod/livinglove/methods", f ); gather( "e:/mindprod/money", f ); // pads gather( "e:/mindprod/phone", f ); gather( "e:/mindprod/politics", f ); gather( "e:/mindprod/politics/laser", f ); // precis gather( "e:/mindprod/projects", f ); gather( "e:/mindprod/quadra", f );
// redir, only makes sense to expand them in place. Don't disturb copied // versions gather( "e:/mindprod/redir", f ); gather( "e:/mindprod/religion", f ); // renney // replicator gather( "e:/mindprod/reunion", f ); // screenshots // snippets // sound gather( "e:/mindprod/utilities", f ); gather( "e:/mindprod/webstarts", f ); // zips
// all files are ready to go.
}
/** * Expands macros written in Java in HTML files files. * * @param args * names of files to process, no wildcards. */ public static void main ( String[] args ) { System.out.println( "\nExpanding macros on entire Mindprod website" );
Tools.setWebRoot( "e:\\mindprod" ); gatherFiles();
System.out.println( gatherCount + " files gathered for mindprod site macro expansion." );
assert gatherCount <= ESTIMATED_TOTAL_FILES; for ( String fileToProcess : filesToProcess ) { assert fileToProcess != null; assert new File( fileToProcess ).exists() : "file to process missing"; try { expandMacrosInFile( new File( fileToProcess ), true /* quiet */); } catch ( Exception e ) { System.err.println( e.getMessage() + " in file " + fileToProcess ); System.err.println(); e.printStackTrace(); // we don't exit. We keep on processing files. } } // end for each file loop System.out.println( "done" ); // don't remove the System.exit( 0 ). It bypasses a Sun bug. System.exit( 0 ); } // end main
} // end ReplaceMindProd
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Stefan Ram - 26 Jan 2006 03:12 GMT You could use Thread.activeCount() to confirm that there is only one thread at the end.
>Tools.setWebRoot( "e:\\mindprod" ); (...) >gatherFiles(); (...) >for ( String fileToProcess : filesToProcess ) (...) >expandMacrosInFile( new File( fileToProcess ) Comment out (hide) the call of "expandMacrosInFile" and see whether the behavior still occurs.
If yes, remove "gatherFiles" and see whether ...
If no: The behavior might have been cause by something in "expandMacrosInFile": So now, edit this method, commenting out its final half or so.
Recursively repeat this until the cause is found.
(This approach is not always possible or might not always work.)
opalpa@gmail.com opalinski from opalpaweb - 26 Jan 2006 20:02 GMT Good day Roedy,
I empathize with this situation; I share an aversion for System.exit . Some ideas:
There are situations where this occurs according to definition in non-gui programs. For example when using a JMF player one needs to stop(), deallocate(), and close() before graceful exit. Perhaps you are inadvartanly and indirectly creating some event loop? --- I know one program that exits gracefully after about ten seconds. Have you waited this long? The general rule is if there is a thread that only wakes up every X seconds then you might wait upto X seconds for it to return from its run(). --- I believe Java spec says that JVM can choose to exit if only demon threads areleft, but does not have to. I doubt this is the case because Sun's jvm always seems to take the exit choice within a couple of seconds. --- Perhaps you should view running threads using JConsole. You will also see each thread's state. There is a negative in that you will see threads related to JConsole listed; this negative is not that large.
Hopefully this email is of some service
All the best, Opalinski opalpa@gmail.com http://www.geocities.com/opalpaweb/
P.S. Jconsole takes a total of maybe five minutes to learn. Run your executable with -Dcom.sun.management.jmxremote and seperately execute jconsole found within java's bin directory. That's it
Here is some documenation: http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html
Roedy Green - 26 Jan 2006 22:11 GMT On 26 Jan 2006 12:02:00 -0800, "opalpa@gmail.com opalinski from opalpaweb" <opalpa@gmail.com> wrote, quoted or indirectly quoted someone who said :
>There are situations where this occurs according to definition in >non-gui programs. For example when using a JMF player one needs to >stop(), deallocate(), and close() before graceful exit. Perhaps you >are inadvartanly and indirectly creating some event loop? This is weird. I am doing pure batch text processing. There are no events or listeners in the whole program. I don't spawn any threads -- any of the usual suspects in refusal to shut down.
Keep going. I hope I have overlooked something.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
opalpa@gmail.com opalinski from opalpaweb - 27 Jan 2006 03:15 GMT > This is weird. I am doing pure batch text processing. Opening files is preceeded by SecurityManager checks. That may involve threads. Stefan Ram suggests a gradual isolation which may be reasonable.
> 6. There is a quantum mechanical effect. Watching the apps with > JConsole makes it work!! Maybe, a long shot, when watching you give extra time which you don't give when not watching and something time limited gets to complete.
Opalinski opalpa@gmail.com http://www.geocities.com/opalpaweb/
Roedy Green - 26 Jan 2006 22:22 GMT On 26 Jan 2006 12:02:00 -0800, "opalpa@gmail.com opalinski from opalpaweb" <opalpa@gmail.com> wrote, quoted or indirectly quoted someone who said :
>I know one program that exits gracefully after about ten seconds. Have >you waited this long? The general rule is if there is a thread that >only wakes up every X seconds then you might wait upto X seconds for it >to return from its run(). There are no timers, execs or threads. About the only exotic feature of this program is using Class.forName to load most of its classes.
Perhaps some sort of incomplete i/o? I do some auxiliary disk processing and perhaps a failure is getting lost, maybe an unclosed file somewhere. Unfortunately Java has no open method you can grep. Also I don't know of a way of testing if and what files are still open.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 26 Jan 2006 23:15 GMT On 26 Jan 2006 12:02:00 -0800, "opalpa@gmail.com opalinski from opalpaweb" <opalpa@gmail.com> wrote, quoted or indirectly quoted someone who said :
>Perhaps you should view running threads using JConsole. You will also >see each thread's state. There is a negative in that you will see >threads related to JConsole listed; this negative is not that large. I'm running it now It turns out to be ten times easier to set up than it first looks. I have documented this, as is my wont, at http://mindprod.com/jgloss/jconsole.html
Surprises for me includes:
1. my "single thread' app had 18 threads.
3. garbage collections are extremely frequent. There over a 1000 of them in the time it took to connect and start looking. However JConsole said that only a few seconds were used in GC. I had assumed they were much less frequent and far more painful.
4. the different aged collections of objects truly do behave differently. the older collections are rock stable. The eden has a two frequency periodicity, like a sound with harmonics.
5. My app uses about 15 MB of heap space by default. I had never done much thinking about ram usage. It is a bit of a shock. In the 1970s I think I would have pulled off an app like this in 200K including the run time.
6. There is a quantum mechanical effect. Watching the apps with JConsole makes it work!!
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Monique Y. Mudama - 27 Jan 2006 00:24 GMT > 6. There is a quantum mechanical effect. Watching the apps with > JConsole makes it work!! That certainly smells like a threading issue. Every time I've used tools to inspect the code and the code started to work, it has always turned out to be a threading problem.
http://catb.org/~esr/jargon/html/H/heisenbug.html
 Signature monique
Ask smart questions, get good answers: http://www.catb.org/~esr/faqs/smart-questions.html
Roedy Green - 27 Jan 2006 02:28 GMT On Thu, 26 Jan 2006 17:24:31 -0700, "Monique Y. Mudama" <spam@bounceswoosh.org> wrote, quoted or indirectly quoted someone who said :
>That certainly smells like a threading issue. Every time I've used >tools to inspect the code and the code started to work, it has always >turned out to be a threading problem. > >http://catb.org/~esr/jargon/html/H/heisenbug.html more weirdness. The bug has now gone altogether. What changed? I added some convenience methods to one of the base classes, but I don't yet use them anywhere.
This is like the old days of FORTRAN when making tiny changes to code changed which code got clobbered when subscripts when out of range.
I don't like this. I have been hit twice in a day with programs that work fine on one compiler but not another. It is a sign Java has grown past the manageable bounds of complexity for a group of humans.
There is no JNI in my code.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
opalpa@gmail.com opalinski from opalpaweb - 27 Jan 2006 03:22 GMT Lack of determinism is a problem. Tread carefully to be certain it's the virtual machine. It won't be first not-exiting bug:
http://developer.java.sun.com/developer/bugParade/bugs/4515058.html http://developer.java.sun.com/developer/bugParade/bugs/4671025.html http://developer.java.sun.com/developer/bugParade/bugs/4465537.html
It appears they do get fixed as Sun agrees jvm should exit gracefully.
Opalinski opalpa@gmail.com http://www.geocities.com/opalpaweb/
Monique Y. Mudama - 27 Jan 2006 04:16 GMT > I don't like this. I have been hit twice in a day with programs that > work fine on one compiler but not another. It is a sign Java has > grown past the manageable bounds of complexity for a group of > humans. Now, now. I happen to know for a fact that they've had problems like this all along. In my graphics class in college, we were using Java 1.1.7. I always made sure to test my projects, which I developed on my windows box, on the lab boxes before submitting them. My professor laughed at me and told me that Java works everywhere.
Well. I was running late for the final project of the semester, and I was running into some weird environment problem, so I turned in the project, a wireframe activated by keypresses, which worked flawlessly on my Windows box. I got a zero on the project because the keypress code didn't work the same between Windows and Solaris. I got a C for the course instead of an A.
I know @!#! well that Java has *never* worked perfectly across all systems.
 Signature monique
Ask smart questions, get good answers: http://www.catb.org/~esr/faqs/smart-questions.html
abmccullough@gmail.com - 26 Jan 2006 16:02 GMT Seems like taking a javacore after the "done" message is printed would a) tell you what thread is active and what it is doing and b) provide the information Sun would need to fix it, if it is truly a bug.
-Andrew
Roedy Green - 26 Jan 2006 23:16 GMT >taking a javacore What class/method is that?
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
abmccullough@gmail.com - 27 Jan 2006 14:31 GMT It's not an API, it's a "dump" of the JVM state that you can use to debug things. It is automatically created in certain error conditions (like a bad signal from JNI code), but you can request one to debug a problem. On Windows, type Ctrl-BREAK in the command window in which the Java.exe is active, on *NIX, issue a kill -3 to the java.exe process. In windows, it will dump to standard out (the console window -- usually, you cannot do this in an IDE) a list of all threads, the currently active thread, held locks and monitors, and a variety of other highly useful information. On *NIX it will write the same information to a file named javacoreTIMESTAMP.txt in the default directory. I use it all the time to determine why things don't stop and other problems. Some web searching should help you find some pointers on interpreting the contents, but the thread dump is pretty straightforward -- it lists all the threads and their current stack.
-Andrew
Roedy Green - 27 Jan 2006 14:45 GMT > On Windows, type Ctrl-BREAK in the command window in which >the Java.exe is active, All I have even seen is a abort. does it hide this dump somewhere without telling you?
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
abmccullough@gmail.com - 27 Jan 2006 17:50 GMT Hmm.. not sure what you mean, but it works for me, I've been doing it for years. You can read a bit more at: http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/index.html
What does "all I have seen is an abort" mean, exactly?
-Andrew
Thomas Hawtin - 26 Jan 2006 19:47 GMT > Without that System.exit, the program keeps on running endlessly using > about 94% of my CPU time after printing "done". [quoted text clipped - 8 lines] > > Any ideas on what could cause such behaviour? Have you tried using jstack (or similar) to see what is continuing to run?
Tom Hawtin
 Signature Unemployed English Java programmer http://jroller.com/page/tackline/
Roedy Green - 26 Jan 2006 23:17 GMT On Thu, 26 Jan 2006 20:01:04 +0000, Thomas Hawtin <usenet@tackline.plus.com> wrote, quoted or indirectly quoted someone who said :
>Have you tried using jstack I had never heard of it till now. Will try that.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 26 Jan 2006 23:33 GMT On Thu, 26 Jan 2006 20:01:04 +0000, Thomas Hawtin <usenet@tackline.plus.com> wrote, quoted or indirectly quoted someone who said :
>Have you tried using jstack (or similar) to see what is continuing to run? Jstack it turns out does not run on windows.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Thomas Hawtin - 27 Jan 2006 00:17 GMT > On Thu, 26 Jan 2006 20:01:04 +0000, Thomas Hawtin > <usenet@tackline.plus.com> wrote, quoted or indirectly quoted someone > who said : > >> Have you tried using jstack (or similar) to see what is continuing to run? > Jstack it turns out does not run on windows. jstack, by default, does much the same as ctrl-\ (which I think is ctrl-break on Windows) only with less fiddling. There are some options, like -m which causes it to NPE lots.
Tom Hawtin
 Signature Unemployed English Java programmer http://jroller.com/page/tackline/
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|