Java Forum / General / July 2009
IOException thrown by process: avoidable?
Ron - 30 Jun 2009 19:31 GMT I'll put my source code and the exception below, but here is an outline and my theory:
I'm using ProcessBuilder to create and start an external process. I'm creating threads to service the i/o in the usual way. Occassionally when reading process output I get an IOException because the stream is closed. I have not closed the stream from the java side.
I am assuming that the stream will also close if the process completes. I think maybe the process finishes, the stream gets closed, my java program attempts to read the next line of output and gets the IOException. I think the timing has to be just right though, because given the same command, I sometimes see the exception and sometimes not. I also thought that maybe the process is not outputing an EOF before exiting as a possible cause, but that doesn't really explain why, for the same command, I sometimes get the exception, and sometimes do not.
Has anyone else observed this before? Is it possible for the stream to get closed because the process has exited? Or should it only close when I explicitly close it in the java program? What do you believe is the root cause of the exception? Is there anyway to avoid the exception in the case where the process has completed and really everything is fine, but still get an exception if the stream closed for other reasons?
java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen (BufferedInputStream.java:145) at java.io.BufferedInputStream.read(BufferedInputStream.java: 308) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) at java.io.InputStreamReader.read(InputStreamReader.java:167) at java.io.BufferedReader.fill(BufferedReader.java:136) at java.io.BufferedReader.readLine(BufferedReader.java:299) at java.io.BufferedReader.readLine(BufferedReader.java:362) at com.nortel.utilities.ProcessUtilities$1.run (ProcessUtilities.java:101)
public static void exec( List<String> command, final List<String> stdOut ) {
ProcessBuilder pb = new ProcessBuilder( command );
final Process p = pb.start();
new Thread() { public void run() { final BufferedReader stdOutReader = new BufferedReader ( new InputStreamReader( p.getInputStream() ), 1024 ); String line; try { while( ( line = stdOutReader.readLine() ) != null ) { if( stdOut != null ) { stdOut.add( line ); } } stdOutReader.close(); } catch( IOException ex ) { Logger.getLogger( ProcessUtilities.class.getName() ).log ( Level.SEVERE, null, ex ); } } }.start();
try { p.waitFor(); } catch( InterruptedException e ) { Thread.currentThread().interrupt(); } }
John B. Matthews - 30 Jun 2009 20:49 GMT In article <fba2615a-7230-4689-a6fd-0c6b1b8995c6@j32g2000yqh.googlegroups.com>,
[...]
> I'm using ProcessBuilder to create and start an external process. > I'm creating threads to service the i/o in the usual way. [quoted text clipped - 4 lines] > I am assuming that the stream will also close if the process > completes. AFAIK, this is not true; you have the close the three streams (stdin, stdout and stderr) explicitly, even if you never use them:
<http://mindprod.com/jgloss/exec.html#GOTCHAS>
> I think maybe the process finishes, the stream gets closed, my java > program attempts to read the next line of output and gets the [quoted text clipped - 4 lines] > why, for the same command, I sometimes get the exception, and > sometimes do not. I cannot explain the variability. Is there any diagnostic output on stderr?
[...]
 Signature John B. Matthews trashgod at gmail dot com <http://sites.google.com/site/drjohnbmatthews>
Ron - 30 Jun 2009 21:12 GMT > In article > <fba2615a-7230-4689-a6fd-0c6b1b899...@j32g2000yqh.googlegroups.com>, [quoted text clipped - 14 lines] > > <http://mindprod.com/jgloss/exec.html#GOTCHAS> Yup. I use Mr. Green's java glossary on a regular basis. In fact, I'm hoping he might even comment on this post (please, please, please :D ).
To keep it brief, I trimmed down my code example. My real exec() function takes a bunch of parameters, and it does the whole redirecting of standard error, closes all three streams, makes sure all the streams get serviced by a thread, etc.
But despite the GOTCHAS section on the mindprod site, I still wonder under what circumstances a stream might close, and whether it could get closed as a result of the process terminating.
BufferedInputStream has the following comment, which makes me think that this is possible. But I'm not an expert, and I'm worried I'm misinterpreting the comment. /** * Atomic updater to provide compareAndSet for buf. This is * necessary because closes can be asynchronous. We use nullness * of buf[] as primary indicator that this stream is closed. (The * "in" field is also nulled out on close.) */
> > I think maybe the process finishes, the stream gets closed, my java > > program attempts to read the next line of output and gets the [quoted text clipped - 7 lines] > I cannot explain the variability. Is there any diagnostic output on > stderr? There is nothing on stderr. Although you cannot see it in the example I posted, I am using a version of exec that services stderr, and there are no messages there. But the external process does what it is supposed to do, and completes successfully, so I'm not surprised there is nothing on stderr.
> [...] > -- > John B. Matthews > trashgod at gmail dot com > <http://sites.google.com/site/drjohnbmatthews> John B. Matthews - 01 Jul 2009 01:56 GMT In article <705241e2-649a-45df-b1e3-d8ad69f3e345@t13g2000yqt.googlegroups.com>,
[...]
> I think maybe the process finishes, the stream gets closed, my > java program attempts to read the next line of output and gets [quoted text clipped - 4 lines] > doesn't really explain why, for the same command, I sometimes get > the exception, and sometimes do not. Looking closer at your initial example, I see that you are creating a new ProcessBuilder and starting it to yield Process p. Then you start a new Thread from which you read p's stdout and wait for p to complete. Instead, shouldn't you do something more like this:
import java.io.BufferedReader; import java.io.InputStreamReader;
public class PBTest {
public static void main(String[] args) { ProcessBuilder pb = new ProcessBuilder("ls", "."); pb.redirectErrorStream(true); try { Process p = pb.start(); String s; // read from the process's combined stdout & stderr BufferedReader stdout = new BufferedReader ( new InputStreamReader(p.getInputStream())); while ((s = stdout.readLine()) != null) { System.out.println(s); } System.out.println("Exit value: " + p.waitFor()); p.getInputStream().close(); p.getOutputStream().close(); p.getErrorStream().close(); } catch (Exception ex) { ex.printStackTrace(System.out); } } }
 Signature John B. Matthews trashgod at gmail dot com <http://sites.google.com/site/drjohnbmatthews>
Ron - 02 Jul 2009 19:44 GMT > In article > <705241e2-649a-45df-b1e3-d8ad69f3e...@t13g2000yqt.googlegroups.com>, [quoted text clipped - 42 lines] > > } The way I did it is recommended in a number of places, including a paper on Sun's website, and also Mr. Green's java glossary. Having dedicated threads to service the standard out, standard error and standard input lets you do stuff with all three simultaneously, and you can treat stdout and stderr separately, if desired. The waitFor() method also gives you the exit value of the process, if that interests you.
But if you don't need any of that, I don't see why your example program would not work. But comparing the two, I don't immediately see a reason why your example and my example would behave differently with regard to the problem I am seeing.
However, I will try it and see whether I actually do get the same issue with your example.
John B. Matthews - 02 Jul 2009 21:33 GMT In article <d9e0655e-0eb5-4a9e-90bd-c7291188cdf7@h2g2000yqg.googlegroups.com>,
> > In article > > <705241e2-649a-45df-b1e3-d8ad69f3e...@t13g2000yqt.googlegroups.com>, [quoted text clipped - 54 lines] > see a reason why your example and my example would behave differently > with regard to the problem I am seeing. I was puzzled by the variability of the problem, which makes me wonder about a threading issue. I see that ProcessBuilder is not synchronized. In particular, "If multiple threads access a ProcessBuilder instance concurrently, and at least one of the threads modifies one of the attributes structurally, it must be synchronized externally." Roedy's example doesn't appear to modify any process attributes, but your code might do so.
<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/ProcessBuilder.html> <http://mindprod.com/jgloss/exec.html>
> However, I will try it and see whether I actually do get the same > issue with your example. I look forward to your results.
 Signature John B. Matthews trashgod at gmail dot com <http://sites.google.com/site/drjohnbmatthews>
Ron - 16 Jul 2009 21:57 GMT > In article > <d9e0655e-0eb5-4a9e-90bd-c7291188c...@h2g2000yqg.googlegroups.com>, [quoted text clipped - 80 lines] > > - Show quoted text - Thanks John Matthews for helping me with this issue.
Sorry for the delay. Today I found some time to do more analysis. In a nutshell, my program generates exceptions, but the one you presented never does. In fact, I can get my program to generate exceptions 100% of the time by adding a sleep statement in the while loop in the thread that is reading the output.
In the multi-threaded example (which may be a necessity, as discussed earlier), the main thread does a waitFor() on the external process. I originally thought that maybe the output streams were getting closed somehow by the external process, but that isn't the case. Once the waitFor() completes, the Process goes out of scope, which causes its streams to get closed.
I have pasted Roedy Green's example below for convenience. There is a comment that starts "The Receiver thread will continue...". Although the Receiver thread continues, the streams is and os get closed when p goes out of scope, so as the thread keeps reading it gets the IOException. If some code is added there to wait for the Receiver thread to finish (not always desirable I understand, but useful for testing), then the exception is avoided.
public static void main( String[] args ) throws IOException { // Boomerang exchoes input to output. final ProcessBuilder pb = new ProcessBuilder( "E:/sys/ boomerang.exe" );
// merge child's error and normal output streams. // Note it is not called setRedirectErrorStream. pb.redirectErrorStream( true );
final Process p = pb.start(); OutputStream os = p.getOutputStream(); InputStream is = p.getInputStream();
// spawn two threads to handle I/O with child while we wait for it to complete. new Thread( new Receiver( is ) ).start(); new Thread( new Sender( os ) ).start(); try { p.waitFor(); } catch ( InterruptedException e ) { Thread.currentThread().interrupt(); } out.println( "Child done" ); // at this point the child is complete. All of its output may or may not have been processed however. // The Receiver thread will continue until it has finished processing it. // You must close the streams even if you never use them! In this case the threads close is and os. p.getErrorStream().close(); } }
Ron - 16 Jul 2009 22:19 GMT > > In article > > <d9e0655e-0eb5-4a9e-90bd-c7291188c...@h2g2000yqg.googlegroups.com>, [quoted text clipped - 142 lines] > > - Show quoted text - But I have more work to do. I just tried Roedy Green's example verbatim, and it works fine, even if a large delay is added to the Receiver thread.
I thought maybe it was because I didn't create a reference to Process p's stream in my thread, but I made that change and tried it, and no improvement. I'll post again when I figure out what I did wrong.
John B. Matthews - 17 Jul 2009 03:31 GMT In article <7c4f1ccc-2e8d-4477-9de4-6315a959c843@h31g2000yqd.googlegroups.com>,
[...]
> > Thanks John Matthews for helping me with this issue. > > [quoted text clipped - 24 lines] > verbatim, and it works fine, even if a large delay is added to the > Receiver thread. Thank you for following up on this. Roedy's example is correct because main() does exit until the Receiver's non-daemon thread exits:
<http://java.sun.com/docs/books/jls/third_edition/html/execution.html#12.8>
If Sender and Receiver must run in separate threads, one approach would be to join() the Receiver's thread after waitFor():
Thread r = new Thread(new Receiver(is)); r.start(); new Thread(new Sender(os)).start(); try { p.waitFor(); r.join(); } catch (InterruptedException e) { ... }
For reference, I test Roedy's example with `tee` as a substitute for Boomerang.
You might also look at Goetz, et al., with reference to the Future interface:
<http://www.javaconcurrencyinpractice.com/>
> I thought maybe it was because I didn't create a reference to Process > p's stream in my thread, but I made that change and tried it, and no > improvement. I'll post again when I figure out what I did wrong.
 Signature John B. Matthews trashgod at gmail dot com <http://sites.google.com/site/drjohnbmatthews>
Ron - 20 Jul 2009 16:55 GMT > In article > <7c4f1ccc-2e8d-4477-9de4-6315a959c...@h31g2000yqd.googlegroups.com>, [quoted text clipped - 64 lines] > > - Show quoted text - Yes, you have it exactly. Thank you.
So just to summarize, in case anyone ever reads this thread looking for help with an issue similar to mine:
In Mr. Green's example, the finalizer (which cleans up Process p and therefore closes the streams) doesn't get called until the service threads complete. So it works great.
But in my program, I'm repeatedly calling a method that kicks off an external process. But then that method finishes and the Process created in that method goes out of scope. The streams associated with that process are therefore closed before the service threads can finish processing the output. The solution is to add a join() as in your example above.
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|