
Signature
Posted via a free Usenet account from http://www.teranews.com
>>> Process process = Runtime.getRuntime().exec(convertCommand);
>>> exitValue = doWaitFor(process);
>> For more control of environment and commandline, you might need
>> ProcessBuilder rather than Runtime.exec().
>What would the advantage be to use ProcessBuilder instead of Runtime?
You get more control over what's actually run - you can give a List<String> or
String[] of params rather than a single String, so you can avoid extra
quoting if the parameters have spaces or special characters in them, you
can set the working directory, and you can control the environment. You
can also set up one ProcessBuilder, and start() it multiple times, if that
fits your use case.
In your simple case, you can also redirectErrorStream(true) before starting
the process, and then you won't have to read the error stream from the
Process.
>Anyway the doWaitFor method (see the appended code in my original post)
>does constantly read - with a sleeper of 500ms - the input and error
>streams of the process...
Right, but polling for output is pretty gross. It's simpler to spin off a
thread to read the input. Hacky sample that should be expanded:
// WARNING: partial and not tested, so errors are likely
builder.setRedirectErrorStream(true);
Process p = builder.start();
final InputStream procStdOut = p.getInputStream();
// start a thread to read the stdout
new Thread("process InputStream handler for " + processName) {
public void run() {
byte[512] buf; // deal with larger blocks than 1 char at a time!
int amtRead;
try {
// read until EOF, blocking when there's nothing to read
while ((amtRead = procStdOut.read(buf) != -1) {
// do something with amtRead bytes of buf
System.out.write(buf, 0, amtRead);
}
} catch (IOException e) {
// shouldn't ever happen for this type of stream
// note failure, queue to retry or whatever you want.
LOGGER.error("impossible IOException!", e);
}
}
}.run();
int exitCode = p.waitFor();
>> Really? Would you provide a link to the bug report?
>http://www.google.ch/search?hl=de&q=java+waitfor+hangs&meta=
>http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=1
These aren't bugs or issues with Process.waitFor(). And they're not win32
specific - it applies to almost all systems.
JavaDoc for java.lang.Process even says "Because some native platforms only
provide limited buffer size for standard input and output streams, failure to
promptly write the input stream or read the output stream of the subprocess
may cause the subprocess to block, and even deadlock."
>> Upload and convert can happen at different times, right?
>If I do understand your question right then no or basically almost no.
Oh. Why not? Video conversion is a slow, expensive process, and you
shouldn't try to do it all at once.
>Let me explain, the file gets uploaded and written to the HD, then
>straight away FFMPEG gets launched. As soon as the process stops, the
>original movie gets deleted from the HD and only the converted one remains.
I'd call that a broken design. It's just plain not going to scale well.
>> Allow upload to
>> some spool directory, and have a separate process (possibly in the same VM,
>> probably not) do the conversion using a pool of converters so that only N
>> are converting at a time.
>
>May you elaborate a bit more on this one?
Sure. Don't process the file right away. Save it to some directory (or DB
blob or whatnot) for incoming files, and have a separate system that
converts the format and deletes the original. That separate system can work
on N files at once, so you can optimize the memory, cpu, and disk usage for
large numbers of requests.
It does mean that you have to show your users a slightly different UI, as they
can now have files queued for processing and files ready for use.
--
Mark Rafn dagon@dagon.net <http://www.dagon.net/>
Vince - 29 Dec 2007 02:55 GMT
Mark first I wanna thank you for your non negligible efforts and time
you did put with your replies...
> ...rather than a single String, so you can avoid extra
> quoting if the parameters have spaces or special characters in them ...
I recall now having used the ProcessBuilder when I was struggling with
the waitFor at one stage but must have come back to the Runtime version
because it didn't solve my problem. With your approach below I gotta
reconsider my code...
> In your simple case, you can also redirectErrorStream(true) before starting
> the process, and then you won't have to read the error stream from the
> Process.
I do not understand this point yet but I'll read about it and get clued up..
> Right, but polling for output is pretty gross.
I thought it wasn't very nice to solve it this way, but good you hammer
the nail further in ;-) As said above I'll reconsider this whole part..
> // WARNING: partial and not tested, so errors are likely
> builder.setRedirectErrorStream(true);
> Process p = builder.start();
I guess at this stage that builder is from type ProcessBuilder...
> final InputStream procStdOut = p.getInputStream();
> // start a thread to read the stdout
[quoted text clipped - 16 lines]
> }.run();
> int exitCode = p.waitFor();
I'll test it and give you a feedback. By just reading the code though,
it all makes sense to me and looks pretty complete..
> Oh. Why not? Video conversion is a slow, expensive process, and you
> shouldn't try to do it all at once.
...
> I'd call that a broken design. It's just plain not going to scale well.
I'm glad to hear that from you. We had long discussions with the
customer and he refuses to queue and thus delay the conversion. The
application is done for regional / International journalists which need
to redact their articles online and thus also need instant access to
their uploaded medias. There is no human process in place between the
time where the journalist releases his article and it gets visible on
the portal. Basically if the journalist prepares his article in word,
quickly copies the content into the portal and pushes the release
button, the article is visible for the public. If the media content
appears now only 10 minutes later because it has been queued for
conversion... no comment...
> ... That separate system can work
> on N files at once, so you can optimize the memory, cpu, and disk usage for
> large numbers of requests.
That's exactly why I'm concerned about the testing and performance... So
basically what can I do?
Thanks Vince

Signature
Posted via a free Usenet account from http://www.teranews.com
Mark Rafn - 29 Dec 2007 08:31 GMT
>> In your simple case, you can also redirectErrorStream(true) before starting
>> the process, and then you won't have to read the error stream from the
>> Process.
>
>I do not understand this point yet but I'll read about it and get clued up..
Every process has output and error output. Normally you have to read both.
If you setRedirectErrorStream(true) before starting it, all the error output
will go to standard output, and you can ignore the error output stream.
>I'm glad to hear that from you. We had long discussions with the
>customer and he refuses to queue and thus delay the conversion.
So if 100 people all upload at the same time, the client would rather delay
them ALL becase they're sharing resources badly rather than processing some
quickly and some more slowly (but not more slowly than all at once).
A bit of reading on queueing theory should convince you that it's pretty much
ALWAYS better to queue than to overextend your resources. Once you're
convinced, you can decide how to convince the client. Or just let him have a
crappy broken app that tries to run hundreds of video transcodes at once if
that's what he demands.
--
Mark Rafn dagon@dagon.net <http://www.dagon.net/>
Vince - 29 Dec 2007 14:43 GMT
> A bit of reading on queueing theory should convince you that it's pretty much
> ALWAYS better to queue than to overextend your resources. Once you're
> convinced, you can decide how to convince the client.
Believe you me I'm convinced since a long time and also tried hardly to
convince the customer but he's as hard as rock. As you said we decided
to let him run into the wall with a conversion process that will become
real slow when multiple user upload their medias. We hope that it might
change his mind at one later stage...
I'm busy writing the test cases but I'll try to implement your code
ASAP... Keep you posted about the result ;-)
Vince

Signature
Posted via a free Usenet account from http://www.teranews.com
Lew - 29 Dec 2007 15:19 GMT
> Believe you me I'm convinced since a long time and also tried hardly to
"Hardly" means "barely at all", so this sentence parses as you "tried barely
at all", which I suspect is the opposite of the intended meaning.
> convince the customer but he's as hard as rock. As you said we decided
> to let him run into the wall with a conversion process that will become
> real slow when multiple user upload their medias. We hope that it might
> change his mind at one later stage...
Nah, they'll just blame you.
I'm being cynical because this has happened to me - I warned a customer that
their decision would make the system unacceptably slow, they harshly overrode
my recommendation, then tried to withhold my fee because the system was
unacceptably slow.

Signature
Lew
Vince - 29 Dec 2007 16:34 GMT
> "Hardly" means "barely at all", so this sentence parses as you "tried
> barely at all", which I suspect is the opposite of the intended meaning.
Thanks pointing on this one, you're right, I meant I tried hard ;-)
> Nah, they'll just blame you.
Obviously, I'm not excepting anything else.. Also a reason why I posted
here..
> then tried to withhold my fee
> because the system was unacceptably slow.
He can try, the platform is running in our data center, I'll power off
his server before he pulls the gun ;-)

Signature
Posted via a free Usenet account from http://www.teranews.com