nio questions

Thread view:

Dave Rathnow - 01 Feb 2006 18:07 GMT

We have an application running on W2K that needs to handle hundreds of
incoming TCP connections at a time. These connections will arrive at a
rate of 200 or more at a time, which means they will all be hitting the
target machine at virtually the same time. This number could go much
higher in a production environment. The initial cut of this program
was not well tested and we are currently having scaling problems.

I've been playing with NIO's non-blocking channels to see how Java will
handle this load. I have a test program that is running on a separate
machine and fires off 200+ threads to attempt connections into the
machine at the same time. What I'm seeing are a lot of "connection
refused" errors yet my server application seems to be idling, that is
there is little CPU usage.

We have a customer who is seeing similar behavior from out existing
application, which runs on a dual 2.4 GHz processor machine with 4 GB
or memory. The machine is virtually idle, yet incoming applications
are getting "connection refused" errors.

So this makes me wonder where is the bottleneck. Is it with the JVM?
The OS? The network interface? Is this type of traffic practical for a
java application to use? Could anyone give me some pointers on how I
much find the source of the problem come up with a solution?'

Thanks
Dave.

Reply to this Message

Thomas Hawtin - 01 Feb 2006 19:51 GMT

> I've been playing with NIO's non-blocking channels to see how Java will
> handle this load. I have a test program that is running on a separate
> machine and fires off 200+ threads to attempt connections into the
> machine at the same time. What I'm seeing are a lot of "connection
> refused" errors yet my server application seems to be idling, that is
> there is little CPU usage.

If all the connection arrive at the same time, it could be just a
question of how many unaccepted connections can be queued. The second
argument in ServerSocket's constructors.

Tom Hawtin

Signature

Unemployed English Java programmer
http://jroller.com/page/tackline/

Reply to this Message

Gordon Beaton - 01 Feb 2006 20:03 GMT

>> fires off 200+ threads to attempt connections into the machine at
>> the same time. What I'm seeing are a lot of "connection refused"
[quoted text clipped - 3 lines]
> question of how many unaccepted connections can be queued. The
> second argument in ServerSocket's constructors.

And if that's the case, then it may help to choose a larger backlog
when creating the ServerSocket (if it's important for the application
to be able to handle peaks such as these).

/gordon

Signature

[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e

Reply to this Message

Dave Rathnow - 01 Feb 2006 21:35 GMT

Thanks for the tip. That seems to have done thet trick, but I have
another problem that is likely caused by something I've missed.

When I accept an incoming connection, I create a SocketChannel,
configure it non-blocking and then register it for OP_READ. I then
enter a loop where I do a select(1000). I assumed that it would pause
for a second or until one or more channels had data waiting. However,
that doesn't seem to be the case. My select returns immediately with a
set of selection keys that says there is data waiting on my channel(s),
but when I issue a read, I get zero length returned. This is cause my
selector thread to chew up all my CPU time and cause problems for other
threads that can't get any CPU time.

I've tried using just "selector.select()" and then calling
selector.wakeup() but my thread sitting in "selector.select()" isn't
waking up.

Have I missed something?

Dave.

Reply to this Message

Gordon Beaton - 02 Feb 2006 07:47 GMT

> When I accept an incoming connection, I create a SocketChannel,
> configure it non-blocking and then register it for OP_READ. I then
[quoted text clipped - 5 lines]
> selector thread to chew up all my CPU time and cause problems for other
> threads that can't get any CPU time.

Off hand it sounds like you've failed to remove each key from the
selected-keys list as you iterate over it, or that you're failing to
test for isReadable() before actually attempting to read. It's hard to
say without seeing the code.

/gordon

Signature

[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e

Reply to this Message

Dave Rathnow - 02 Feb 2006 12:00 GMT

Okay,

You're partially right. I remove all the keys after iterating over the list
of selected keys but I am checking for is readable. Here's what I'm doing

private void waitForSelector() throws IOException {
if (selector.select(500) > 0) {
for (Iterator iter = selector.selectedKeys().iterator();
iter.hasNext();)
performOperationForSelectionKey((SelectionKey)iter.next());
selector.selectedKeys().clear();
}
}

private void performOperationForSelectionKey(SelectionKey selectionKey)
{
if (selectionKey.isConnectable())

((SocketChannelManager)selectionKey.attachment()).finishConnecting();
if (selectionKey.isReadable())

((SocketChannelManager)selectionKey.attachment()).readIncomingData();
if (selectionKey.isWritable())

((SocketChannelManager)selectionKey.attachment()).writeOutgoingData();
}

> > When I accept an incoming connection, I create a SocketChannel,
> > configure it non-blocking and then register it for OP_READ. I then
[quoted text clipped - 12 lines]
>
> /gordon

Reply to this Message

Gordon Beaton - 02 Feb 2006 13:09 GMT

> You're partially right. I remove all the keys after iterating over
> the list of selected keys but I am checking for is readable. Here's
> what I'm doing

[...]

Does calling clear() have the same effect as calling remove() on each
of the elements in the selected-keys set? (I would hope so at any
rate).

Are you by any chance registering OP_WRITE as well? (if so, don't)

Or perhaps you have run into this:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4629307

(There are several other selector-related bugs, I just browsed through
them until I found one that seems to fit.)

/gordon

Signature

[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e

Reply to this Message

Dave Rathnow - 02 Feb 2006 15:07 GMT

Yes, clear does have the same effect and this looks like the bug. The
behavior is exactly the same. The problem seems to be happen only
after I blast the application with many connection. I can't reproduce
it with only a few connections.

Thanks for looking this up! Now all I need to do is find a workaround.
:(

Dave

Reply to this Message

Dave Rathnow - 02 Feb 2006 15:09 GMT

Hmmm...I just realized this was suppose to have been fixed in 1.4.1,
but that's the version I'm using :(

Reply to this Message

Thomas Hawtin - 02 Feb 2006 20:55 GMT

> Hmmm...I just realized this was suppose to have been fixed in 1.4.1,
> but that's the version I'm using :(

Yeah, NIO had a 'troubled' early history. When did 1.4.2 come out?? The
only sensible reason I can see to stick with 1.4 is to avoid newly
introduced bugs. But to use an early version of an old version is nuts.

Tom Hawtin

Signature

Unemployed English Java programmer
http://jroller.com/page/tackline/

Reply to this Message

Gordon Beaton - 02 Feb 2006 16:04 GMT

> Yes, clear does have the same effect and this looks like the bug. The
> behavior is exactly the same. The problem seems to be happen only
> after I blast the application with many connection. I can't reproduce
> it with only a few connections.

In that case, are you sure it isn't this problem instead?
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4503092

/gordon

Signature

[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e

Reply to this Message

nio questions

Free Magazines