Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / July 2009

Tip: Looking for answers? Try searching our database.

IOException thrown by process: avoidable?

Thread view: 
Ron - 30 Jun 2009 19:31 GMT
I'll put my source code and the exception below, but here is an
outline and my theory:

I'm using ProcessBuilder to create and start an external process.  I'm
creating threads to service the i/o in the usual way.  Occassionally
when reading process output I get an IOException  because the stream
is closed.  I have not closed the stream from the java side.

I am assuming that the stream will also close if the process
completes.  I think maybe the process finishes, the stream gets
closed, my java program attempts to read the next line of output and
gets the IOException.  I think the timing has to be just right though,
because given the same command, I sometimes see the exception and
sometimes not.  I also thought that maybe the process is not outputing
an EOF before exiting as a possible cause, but that doesn't really
explain why, for the same command, I sometimes get the exception, and
sometimes do not.

Has anyone else observed this before?  Is it possible for the stream
to get closed because the process has exited?  Or should it only close
when I explicitly close it in the java program?  What do you believe
is the root cause of the exception?  Is there anyway to avoid the
exception in the case where the process has completed and really
everything is fine, but still get an exception if the stream closed
for other reasons?

java.io.IOException: Stream closed
       at java.io.BufferedInputStream.getBufIfOpen
(BufferedInputStream.java:145)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:
308)
       at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
       at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
       at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
       at java.io.InputStreamReader.read(InputStreamReader.java:167)
       at java.io.BufferedReader.fill(BufferedReader.java:136)
       at java.io.BufferedReader.readLine(BufferedReader.java:299)
       at java.io.BufferedReader.readLine(BufferedReader.java:362)
       at com.nortel.utilities.ProcessUtilities$1.run
(ProcessUtilities.java:101)

  public static void exec( List<String> command,
                                    final List<String> stdOut )
  {

     ProcessBuilder pb = new ProcessBuilder( command );

     final Process p = pb.start();

     new Thread()
     {
        public void run()
        {
           final BufferedReader stdOutReader = new BufferedReader
( new InputStreamReader( p.getInputStream() ), 1024 );
           String line;
           try
           {
              while( ( line = stdOutReader.readLine() ) != null )
              {
                 if( stdOut != null )
                 {
                    stdOut.add( line );
                 }
              }
              stdOutReader.close();
           }
           catch( IOException ex )
           {
              Logger.getLogger( ProcessUtilities.class.getName() ).log
( Level.SEVERE, null, ex );
           }
        }
     }.start();

     try
     {
        p.waitFor();
     }
     catch( InterruptedException e )
     {
        Thread.currentThread().interrupt();
     }
  }
John B. Matthews - 30 Jun 2009 20:49 GMT
In article
<fba2615a-7230-4689-a6fd-0c6b1b8995c6@j32g2000yqh.googlegroups.com>,

[...]
> I'm using ProcessBuilder to create and start an external process.  
> I'm creating threads to service the i/o in the usual way.  
[quoted text clipped - 4 lines]
> I am assuming that the stream will also close if the process
> completes.

AFAIK, this is not true; you have the close the three streams (stdin,
stdout and stderr) explicitly, even if you never use them:

<http://mindprod.com/jgloss/exec.html#GOTCHAS>

> I think maybe the process finishes, the stream gets closed, my java
> program attempts to read the next line of output and gets the
[quoted text clipped - 4 lines]
> why, for the same command, I sometimes get the exception, and
> sometimes do not.

I cannot explain the variability. Is there any diagnostic output on
stderr?

[...]
Signature

John B. Matthews
trashgod at gmail dot com
<http://sites.google.com/site/drjohnbmatthews>

Ron - 30 Jun 2009 21:12 GMT
> In article
> <fba2615a-7230-4689-a6fd-0c6b1b899...@j32g2000yqh.googlegroups.com>,
[quoted text clipped - 14 lines]
>
> <http://mindprod.com/jgloss/exec.html#GOTCHAS>

Yup.  I use Mr. Green's java glossary on a regular basis.  In fact,
I'm hoping he might even comment on this post (please, please,
please :D ).

To keep it brief, I trimmed down my code example.  My real exec()
function takes a bunch of parameters, and it does the whole
redirecting of standard error, closes all three streams, makes sure
all the streams get serviced by a thread, etc.

But despite the GOTCHAS section on the mindprod site, I still wonder
under what circumstances a stream might close, and whether it could
get closed as a result of the process terminating.

BufferedInputStream has the following comment, which makes me think
that this is possible.  But I'm not an expert, and I'm worried I'm
misinterpreting the comment.
   /**
    * Atomic updater to provide compareAndSet for buf. This is
    * necessary because closes can be asynchronous. We use nullness
    * of buf[] as primary indicator that this stream is closed. (The
    * "in" field is also nulled out on close.)
    */

> > I think maybe the process finishes, the stream gets closed, my java
> > program attempts to read the next line of output and gets the
[quoted text clipped - 7 lines]
> I cannot explain the variability. Is there any diagnostic output on
> stderr?

There is nothing on stderr.  Although you cannot see it in the example
I posted, I am using a version of exec that services stderr, and there
are no messages there.  But the external process does what it is
supposed to do, and completes successfully, so I'm not surprised there
is nothing on stderr.

> [...]
> --
> John B. Matthews
> trashgod at gmail dot com
> <http://sites.google.com/site/drjohnbmatthews>
John B. Matthews - 01 Jul 2009 01:56 GMT
In article
<705241e2-649a-45df-b1e3-d8ad69f3e345@t13g2000yqt.googlegroups.com>,

[...]
> I think maybe the process finishes, the stream gets closed, my
> java program attempts to read the next line of output and gets
[quoted text clipped - 4 lines]
> doesn't really explain why, for the same command, I sometimes get
> the exception, and sometimes do not.

Looking closer at your initial example, I see that you are creating a
new ProcessBuilder and starting it to yield Process p. Then you start a
new Thread from which you read p's stdout and wait for p to complete.
Instead, shouldn't you do something more like this:

import java.io.BufferedReader;
import java.io.InputStreamReader;

public class PBTest {

   public static void main(String[] args) {
       ProcessBuilder pb = new ProcessBuilder("ls", ".");
       pb.redirectErrorStream(true);
       try {
           Process p = pb.start();
           String s;
           // read from the process's combined stdout & stderr
           BufferedReader stdout = new BufferedReader (
               new InputStreamReader(p.getInputStream()));
           while ((s = stdout.readLine()) != null) {
               System.out.println(s);
           }
           System.out.println("Exit value: " + p.waitFor());
           p.getInputStream().close();
           p.getOutputStream().close();
           p.getErrorStream().close();
        } catch (Exception ex) {
           ex.printStackTrace(System.out);
       }
   }
}

Signature

John B. Matthews
trashgod at gmail dot com
<http://sites.google.com/site/drjohnbmatthews>

Ron - 02 Jul 2009 19:44 GMT
> In article
> <705241e2-649a-45df-b1e3-d8ad69f3e...@t13g2000yqt.googlegroups.com>,
[quoted text clipped - 42 lines]
>
> }

The way I did it is recommended in a number of places, including a
paper on Sun's website, and also Mr. Green's java glossary.  Having
dedicated threads to service the standard out, standard error and
standard input lets you do stuff with all three simultaneously, and
you can treat stdout and stderr separately, if desired.  The waitFor()
method also gives you the exit value of the process, if that interests
you.

But if you don't need any of that, I don't see why your example
program would not work.  But comparing the two, I don't immediately
see a reason why your example and my example would behave differently
with regard to the problem I am seeing.

However, I will try it and see whether I actually do get the same
issue with your example.
John B. Matthews - 02 Jul 2009 21:33 GMT
In article
<d9e0655e-0eb5-4a9e-90bd-c7291188cdf7@h2g2000yqg.googlegroups.com>,

> > In article
> > <705241e2-649a-45df-b1e3-d8ad69f3e...@t13g2000yqt.googlegroups.com>,
[quoted text clipped - 54 lines]
> see a reason why your example and my example would behave differently
> with regard to the problem I am seeing.

I was puzzled by the variability of the problem, which makes me wonder
about a threading issue. I see that ProcessBuilder is not synchronized.
In particular, "If multiple threads access a ProcessBuilder instance
concurrently, and at least one of the threads modifies one of the
attributes structurally, it must be synchronized externally." Roedy's
example doesn't appear to modify any process attributes, but your code
might do so.

<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/ProcessBuilder.html>
<http://mindprod.com/jgloss/exec.html>

> However, I will try it and see whether I actually do get the same
> issue with your example.

I look forward to your results.

Signature

John B. Matthews
trashgod at gmail dot com
<http://sites.google.com/site/drjohnbmatthews>

Ron - 16 Jul 2009 21:57 GMT
> In article
> <d9e0655e-0eb5-4a9e-90bd-c7291188c...@h2g2000yqg.googlegroups.com>,
[quoted text clipped - 80 lines]
>
> - Show quoted text -

Thanks John Matthews for helping me with this issue.

Sorry for the delay.  Today I found some time to do more analysis.  In
a nutshell, my program generates exceptions, but the one you presented
never does.  In fact, I can get my program to generate exceptions 100%
of the time by adding a sleep statement in the while loop in the
thread that is reading the output.

In the multi-threaded example (which may be a necessity, as discussed
earlier), the main thread does a waitFor() on the external process.  I
originally thought that maybe the output streams were getting closed
somehow by the external process, but that isn't the case.  Once the
waitFor() completes, the Process goes out of scope, which causes its
streams to get closed.

I have pasted Roedy Green's example below for convenience.  There is a
comment that starts "The Receiver thread will continue...".  Although
the Receiver thread continues, the streams is and os get closed when p
goes out of scope, so as the thread keeps reading it gets the
IOException.  If some code is added there to wait for the Receiver
thread to finish (not always desirable I understand, but useful for
testing), then the exception is avoided.

   public static void main( String[] args ) throws IOException
       {
       // Boomerang exchoes input to output.
       final ProcessBuilder pb = new ProcessBuilder( "E:/sys/
boomerang.exe" );

       // merge child's error and normal output streams.
       // Note it is not called setRedirectErrorStream.
       pb.redirectErrorStream( true );

       final Process p = pb.start();
       OutputStream os = p.getOutputStream();
       InputStream is = p.getInputStream();

       // spawn two threads to handle I/O with child while we wait
for it to complete.
       new Thread( new Receiver( is ) ).start();
       new Thread( new Sender( os ) ).start();
       try
           {
           p.waitFor();
           }
       catch ( InterruptedException e )
           {
           Thread.currentThread().interrupt();
           }
       out.println( "Child done" );
       // at this point the child is complete.  All of its output may
or may not have been processed however.
       // The Receiver thread will continue until it has finished
processing it.
       // You must close the streams even if you never use them!  In
this case the threads close is and os.
       p.getErrorStream().close();
       }
   }
Ron - 16 Jul 2009 22:19 GMT
> > In article
> > <d9e0655e-0eb5-4a9e-90bd-c7291188c...@h2g2000yqg.googlegroups.com>,
[quoted text clipped - 142 lines]
>
> - Show quoted text -

But I have more work to do.  I just tried Roedy Green's example
verbatim, and it works fine, even if a large delay is added to the
Receiver thread.

I thought maybe it was because I didn't create a reference to Process
p's stream in my thread, but I made that change and tried it, and no
improvement.  I'll post again when I figure out what I did wrong.
John B. Matthews - 17 Jul 2009 03:31 GMT
In article
<7c4f1ccc-2e8d-4477-9de4-6315a959c843@h31g2000yqd.googlegroups.com>,

[...]
> > Thanks John Matthews for helping me with this issue.
> >
[quoted text clipped - 24 lines]
> verbatim, and it works fine, even if a large delay is added to the
> Receiver thread.

Thank you for following up on this. Roedy's example is correct because
main() does exit until the Receiver's non-daemon thread exits:

<http://java.sun.com/docs/books/jls/third_edition/html/execution.html#12.8>

If Sender and Receiver must run in separate threads, one approach would be
to join() the Receiver's thread after waitFor():

       Thread r = new Thread(new Receiver(is));
       r.start();
       new Thread(new Sender(os)).start();
       try {
           p.waitFor();
           r.join();
       } catch (InterruptedException e) { ... }

For reference, I test Roedy's example with `tee` as a substitute for
Boomerang.

You might also look at Goetz, et al., with reference to the Future
interface:

<http://www.javaconcurrencyinpractice.com/>

> I thought maybe it was because I didn't create a reference to Process
> p's stream in my thread, but I made that change and tried it, and no
> improvement.  I'll post again when I figure out what I did wrong.

Signature

John B. Matthews
trashgod at gmail dot com
<http://sites.google.com/site/drjohnbmatthews>

Ron - 20 Jul 2009 16:55 GMT
> In article
> <7c4f1ccc-2e8d-4477-9de4-6315a959c...@h31g2000yqd.googlegroups.com>,
[quoted text clipped - 64 lines]
>
> - Show quoted text -

Yes, you have it exactly.  Thank you.

So just to summarize, in case anyone ever reads this thread looking
for help with an issue similar to mine:

In Mr. Green's example, the finalizer (which cleans up Process p and
therefore closes the streams) doesn't get called until the service
threads complete.  So it works great.

But in my program, I'm repeatedly calling a method that kicks off an
external process.  But then that method finishes and the Process
created in that method goes out of scope.  The streams associated with
that process are therefore closed before the service threads can
finish processing the output.  The solution is to add a join() as in
your example above.


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2010 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.