> I've got a class which sends multipart MIME data as a POST request to
> a web server. It works fine with small files, but larger files (still
[quoted text clipped - 6 lines]
> caching the data being sent, and I really can't see why the memory
> consumption would be related to file size.
Maybe it's storing the file in mem to be able to write a content-size
header. For serious application I'd always use another HTTP client such
as the one from apache.
http://jakarta.apache.org/commons/httpclient/
> I've found that if I don't write the file data to the OutputStream
> (comment out outStream.write(bytes,0,a);) then it completes OK so it
[quoted text clipped - 63 lines]
> }
> }
Btw, I suggest you rethink the way you use exceptions. Either catch them
at the end of the method or declare them as thrown but catching them in
the middle and continuing doesn't seem a good way to deal with failure
IMHO.
Kind regards
robert
Simon Andrews - 06 Jan 2006 12:19 GMT
>>I've got a class which sends multipart MIME data as a POST request to
>>a web server. It works fine with small files, but larger files (still
[quoted text clipped - 7 lines]
> as the one from apache.
> http://jakarta.apache.org/commons/httpclient/
That would make sense (even though I think the content-size is optional
for these kinds of requests. I'll look into Jakarta - cheers!
> Btw, I suggest you rethink the way you use exceptions. Either catch them
> at the end of the method or declare them as thrown but catching them in
> the middle and continuing doesn't seem a good way to deal with failure
> IMHO.
The code posted was a hacked together version of what's actually used in
the script (which is a lot better controlled but contains a lot of
irrelevant stuff) - the exception handling here is the automated stuff
eclipse adds just to allow it to compile :-)
Thanks again
Simon.
Robert Klemme - 06 Jan 2006 14:32 GMT
>>> I've got a class which sends multipart MIME data as a POST request
>>> to a web server. It works fine with small files, but larger files
[quoted text clipped - 11 lines]
> optional for these kinds of requests. I'll look into Jakarta -
> cheers!
Yeah, but the HTTP implementation of the JDK is quite rudimentary. Even
allows for proxy usage only via an ugly hack...
>> Btw, I suggest you rethink the way you use exceptions. Either catch
>> them at the end of the method or declare them as thrown but catching
[quoted text clipped - 5 lines]
> irrelevant stuff) - the exception handling here is the automated stuff
> eclipse adds just to allow it to compile :-)
My eclipse lets me choose wherer I want a try-catch block or a throws
declaration. :-)
> Thanks again
You're welcome!
Kind regards
robert
Simon Andrews - 06 Jan 2006 13:19 GMT
>>I've got a class which sends multipart MIME data as a POST request to
>>a web server. It works fine with small files, but larger files (still
>><50Mb) cause it to throw an OutOfMemory exception:
>
> Maybe it's storing the file in mem to be able to write a content-size
> header.
After a bit more playing this led me to what actually happens.
In fact the HttpURLConnection doesn't calculate a content-size for you
(you can set one yourself, but it's not checked) - but it does cache all
of the data sent to it. For some reason it doesn't actually send any
data until you either call getInputStream() or getResposeCode() - no
matter how much data you've sent to the OutputStream it never leaves the
client until one of those methods is called. This makes it useless for
sending large POST requests.
I'd like to think that there was a way to override this so that the
caching is removed and the data is sent as soon as it's written - but I
couldn't find one. If anyone can tell me how to prevent this caching
I'd certainly appreciate it!
Cheers
Simon.
Svante Frey - 06 Jan 2006 14:39 GMT
> In fact the HttpURLConnection doesn't calculate a content-size for you
> (you can set one yourself, but it's not checked) - but it does cache all
[quoted text clipped - 8 lines]
> couldn't find one. If anyone can tell me how to prevent this caching
> I'd certainly appreciate it!
Sun has added this possiblity in Java 1.5: have a look at the
documentation for HttpUrlConnection.setChunkedStreamingMode . Don't
know if there are any possiblities to do it in earlier Java versions.
Also see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5026745 and
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4212479 for some
discussion of this problem.
Robert Klemme - 06 Jan 2006 14:53 GMT
>>> I've got a class which sends multipart MIME data as a POST request
>>> to a web server. It works fine with small files, but larger files
[quoted text clipped - 17 lines]
> I couldn't find one. If anyone can tell me how to prevent this
> caching I'd certainly appreciate it!
Just create a thread that reads all content from InputStream and discards
it.
robert