A few quick bits of advice:
1) use the second of your suggested approaches. cleaner that way.
2) use Apache MINA to implement the client and server in a structured
way, without a lot of the hassle
3) make the handling of individual protocol messages stateless
whenever possible. easier to debug.

T

I view implementing a protocol as related to implementing a parser
for a language.  When writing a parser, you can do a recursive-descent
parser (which is like the first form above) or you can use a state
machine (like the second form above).  Which one is best tends to
depend on the specifics (and that applies to both parsing and network
protocols).

If your protocol only has a small number of possible states, then the
state machine is probably a good choice.  If your protocol has a lot
of states, then mapping the states to the call stack (i.e. structuring
it like a recursive-descent parser) might provide a natural way to
keep things structured and clear.  Keep in mind, too, that exceptions
could be useful for handling errors, like an unexpected disconnect,
which is something that could happen at any point in time with a
connection-oriented protocol.  That could be an argument for mapping
the protocol state to a thread's call stack.

Note that I mentioned threads.  The style of mapping protocol state
to the call stack doesn't make a lot of sense unless each thread
handles exactly one session at a time.  So the decision also
depends on your I/O and threading model.

Note that the best choice also may depend on the specifics of the
protocol itself.  Is it a protocol which follows a question, answer,
question, answer, question, answer pattern and each side never
sends a new message while it's waiting for a response?  Or is it
a protocol where either side could send a message at any time
and there might possibly be multiple logical requests or activities
going on at once?  In the alternating question/answer case, you
can easily map that onto a call stack, but in the latter case,
trying to fit that structure to the call stack will probably only
make the program *more* confusing.

  - Logan

From what I can tell, most complex protocols use some sort of state
machine. A few network protocol RFCs show an overview diagram as if it
were a finite state machine (in ASCII art, as well!).

IIUC, I've heard pipelining used for these types of responses: the
requests are queued up and all sent and then wait for their responses.
Setting up a queue of responses and then processing them in turn would
be the way to go here.