java.net.SocketException + too many open files

Thread view:

bond - 20 Nov 2007 19:46 GMT

Hi,

I am using URLConnecton(wrapped with HttpURLConnection) class to
download images and store into file system from a HTTP source.
If the number of images exceeds 10K, i get java.net.SocketException
and "too many open files" error.
I do close the inputstream and diconnect the connection after i read
each image.
Note: this happens only on Linux

Can any one give me any fix for this?

thanks in advance,

Reply to this Message

Daniel Pitts - 20 Nov 2007 19:52 GMT

> Hi,
>
[quoted text clipped - 9 lines]
>
> thanks in advance,

It might be useful to do a heap dump to see what objects are still lying
around. You might find that you aren't actually closing all streams/files.

You might also want to change your ulimits, although I'm not sure
that'll help in this case.

Signature

Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>

Reply to this Message

Gordon Beaton - 20 Nov 2007 19:53 GMT

> I am using URLConnecton(wrapped with HttpURLConnection) class to
> download images and store into file system from a HTTP source. If
[quoted text clipped - 3 lines]
> each image.
> Note: this happens only on Linux

There are other places your application might be opening files than
just URLConnection... for example every Runtime.exec() costs 3 open
files.

To check the limit, use "ulimit -a" or "ulimit -n" in the shell.

This isn't a solution, but you can increase the limit by running
"ulimit -n someNumber" in the shell before running the application, or
put the command in your .bash_profile or similar place so it runs
every time you login.

The solution is to find out what files your application isn't closing,
and make sure you close them. You can do that by doing "ls -l" in
/proc/NN/fd, or running lsof on the process.

/gordon

--

Reply to this Message

java.net.SocketException + too many open files

Free Magazines