NIO - OutOfMemoryException

Thread view:

Camille - 15 Nov 2005 21:30 GMT

Hi,
first of all, i'm not sure of the newsgroup to post this message.
If i'm not in the good place, please tell me.

I'm using NIO in a client server project, and from a sudden, I get
OutOfMemoryException.
After some search on the web, I found that ByteBuffer.allocateDirect
was often a problem if the heap size was too low.

BUT, the problem is that I get this exception when writing on NIO, not
when reading.

Here's the code :

FIFOQueue m_responsesQueue = new FIFOQueue();
...
AKSResponse response = null;
SocketChannel client;
Charset charset = Charset.forName("UTF-8"); // client encoding

while (m_responsesQueue.size() > 0)
{
response = (AKSResponse) m_responsesQueue.getFirst();
client = response.getChannel();
if (client.isOpen())
{
if (!client.socket().isClosed())
{
if (!client.socket().isOutputShutdown())
{
client.write(charset.encode(response.getData()));
}
}
}
response.free();
response = null;
m_responsesQueue.dequeue();
}

And I get the following :

java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:285)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:54)
at sun.nio.ch.IOUtil.write(IOUtil.java:69)
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:300)
at aks.gameServer.net.AKSWriteDaemon.run
(AKSWriteDaemon.java:102)

The line 102 being the client.write(charset.......

I presume I have to allocate more memory to my VM with the Xmx, Xmn,
X... settings, but I'm confused with all the possibilites : what about
XNewSize, XMaxNewSize, Xms, ...

Thanks in advance for your help,

Camille

Reply to this Message

Thomas Hawtin - 16 Nov 2005 00:51 GMT

> first of all, i'm not sure of the newsgroup to post this message.
> If i'm not in the good place, please tell me.

Looks good to me.

> I'm using NIO in a client server project, and from a sudden, I get
> OutOfMemoryException.
> After some search on the web, I found that ByteBuffer.allocateDirect
> was often a problem if the heap size was too low.

> java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method)
[quoted text clipped - 6 lines]
> at aks.gameServer.net.AKSWriteDaemon.run
> (AKSWriteDaemon.java:102)

The code appears to cache direct buffers, if you are use non-direct
buffers. Possibly there is a significant delay between SoftReferences
getting cleared and reference handler cleansing the buffer. If you are
on Linux or Solaris, you might see the problem disappear when running as
root (I believe thread priorities actually work then). I don't know the
details of interaction between SoftReferences, finalisable objects, the
garbage collector and the reference handler. If it requires another
appropriate GC cycle to queue the finaliser after the SoftReference is
cleared, then that may well cause problems.

If you have many threads, then you can end up allocating a silly number
of these temporary direct buffers, as the cache is thread-local. Their
interaction with the garbage collector is such that you don't want to be
doing that.

The obvious solution is to use directly allocated buffers yourself. But
be sure they are long lived, rather than constantly allocated and collected.

> I presume I have to allocate more memory to my VM with the Xmx, Xmn,
> X... settings, but I'm confused with all the possibilites : what about
> XNewSize, XMaxNewSize, Xms, ...

In general you should set -Xmx to how much heap you want. Set the
minimum heap, -Xms, to the same in order to avoid unnecessary GC. If you
don't want to do that, I believe using -server changes SoftReference
calculations to use free heap space if maximum were allocated in place
of current free heap space. IIRC, there is a magic -XX option to change
the maximum space allocable for direct buffers, as a fraction of heap size.

Tom Hawtin

Signature

Unemployed English Java programmer
http://jroller.com/page/tackline/

Reply to this Message

camille.chafer@gmail.com - 05 Dec 2005 19:56 GMT

Hi folks,
Just a little message to say I finnaly found the solution.
The problem was the SocketChannel.write implementation.
Indeed, even if you don't use DirectByteBuffer in your code,
SocketChannel.write does it for you.
This means that every time you call the write method, a new
DirectByteBuffer is created. And since there is no real and effective
GC on this type of objects, on heavy load, your application simply
crash.

The solution ? I did create a big DirectBuffer by myself, copy the
encoded result into it, then use it to do the write stuff.
Here's the code :

ByteBuffer outputDirectBuffer =
ByteBuffer.allocateDirect(OUTPUT_DIRECTBUFFER_SIZE);
int dataSent, dataToSend;
while (m_responsesQueue.size() > 0) {
    response = (AKSResponse) m_responsesQueue.getFirst();
client = response.getChannel();
if (client.isOpen()) {
if (!client.socket().isClosed()) {
if (!client.socket().isOutputShutdown()) {
                outputDirectBuffer.clear();
                outputDirectBuffer.put(charset.encode(response.getData()));
                outputDirectBuffer.flip();
                dataToSend = outputDirectBuffer.limit() -
outputDirectBuffer.position();
                dataSent = client.write(outputDirectBuffer);
                if (dataSent != dataToSend) {
                    //.. not enough bytes were sent. Maybe the network heap full ?
                    logger.fatal ("Unable to send " + dataToSend + " bytes : " +
dataSend + " sent");
     }
}
}
}
response.free();
response = null;
m_responsesQueue.dequeue();
}

Using this, I now achieve to let about 6.000 players join games on my
server, without any memory problem).
Thanks again for your help,

Camille

Reply to this Message

NIO - OutOfMemoryException

Free Magazines