 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / General / March 2008Shell out without so many threads?
Long ago I carefully implemented a Shell out command, following the guidance from this: http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html Summary: use threads to gather stdout and stderr from your exec, otherwise it might deadlock. I even added (yet another) thread to implement timeouts (to kill a shell script gone infinite). Time moves on, and I now, under Tomcat, have a secondary problem. My threads are thrashing. I'm using 4 threads (main, stdout_gather, stderr_gather, timeout) per shell, and it's just too many. Does anybody have any advice, experience, or code to reduce this number. My (hazy) thoughts involve using non-blocking IO to pick up multiple stdout/stderr streams in a single thread (but I can't get a Selectable from a Stream) and using a single thread with a "next timeout" model to use a single timout thread for all shell outs. All input gratefully received. BugBear > Long ago I carefully implemented a Shell out > command, following the guidance from this: [quoted text clipped - 27 lines] > > BugBear If you use ProcessBuilder you can merge stdout and stderr, that will save you 25% of your threads. See ProcessBuilder.redirectErrorStream(boolean redirect).  SignatureKnute Johnson email s/nospam/linux/ > Long ago I carefully implemented a Shell out > command, following the guidance from this: [quoted text clipped - 27 lines] > > BugBear Define crashing. Usually the JVM can deal with many thousands of threads. The JVM gets away with this by not having a 1:1 mapping to native threads on systems where they're expensive. The primary limiting factor is how much memory you're holding on to when the thread blocks. You may need to null unused local variables before blocking to get past a few hundred threads. Check the thread group when you're running under Tomcat. Some thread groups shouldn't be interfered with. SignatureI don't read Google's spam. Reply with another service. >> Long ago I carefully implemented a Shell out >> command, following the guidance from this: [quoted text clipped - 29 lines] > > Define crashing. "thrashing"; high kernel load on Solaris (as indicated by Solaris performance monitoring tools) traced to process swapping in the JVM. > Usually the JVM can deal with many thousands of threads. Interesting; the "standard" config file for tomcat has a "max" of 150. > The JVM gets > away with this by not having a 1:1 mapping to native threads on systems [quoted text clipped - 4 lines] > Check the thread group when you're running under Tomcat. Some thread > groups shouldn't be interfered with. Pardon my ignorance; that's a "thread group" BugBear >> Check the thread group when you're running under Tomcat. Some thread >> groups shouldn't be interfered with. > > Pardon my ignorance; that's a "thread group" Bad typo! what's a "thread group" ? BugBear >>> Check the thread group when you're running under Tomcat. Some thread >>> groups shouldn't be interfered with. [quoted text clipped - 6 lines] > > BugBear A group of threads, as implemented by java.lang.ThreadGroup. They'll have some higher-order cohesiveness, e.g., grouping a series of worker threads together. SignatureBeware of bugs in the above code; I have only proved it correct, not tried it. -- Donald E. Knuth > >> Check the thread group when you're running under Tomcat. Some thread > >> groups shouldn't be interfered with. [quoted text clipped - 6 lines] > > BugBear The Tomcat thread group is a ThreadGroup instance. It makes it easier to act on many threads at once. The other thing to check is the native options: -Xss<size> Stack size per thread. -XX:+UseBoundThreads Forces 1:1 thread ratio. Probably not what you want. -XX:-UseLWPSynchronization SignatureI don't read Google's spam. Reply with another service. > Usually the JVM can deal with many thousands of threads. The JVM gets > away with this by not having a 1:1 mapping to native threads on systems > where they're expensive. Are any JVM's besides older JRockit's with -Xthinthreads actually doing that ? Arne On Wed, 19 Mar 2008 16:16:48 +0000, bugbear <bugbear@trim_papermule.co.uk_trim> wrote, quoted or indirectly quoted someone who said : >Does anybody have any advice, experience, or code >to reduce this number. some possibilities: 1. write the shelled out code in Java and just call it directly. 2. tell the shelled out code to use files for in/out.. Prepare the input file ahead of time and read the output file after termination. 3. tell the shelled out code to communicate via transactions that look just like requests coming in over the web. see http://mindprod.com/jgloss/http.html 4. create a JNI hook to the shelled out code. see http://mindprod.com/jgloss/jni.html 5. create a queue to a thread pool so that you don't spawn too many shells at once. see http://mindprod.com/jgloss/queue.html SignatureRoedy Green Canadian Mind Products The Java Glossary http://mindprod.com Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.