I can't help directly, but once managed to do something somewhat similar.

I have a Windows application that links the JVM in the same kind of way as
Matlab and I've managed to get the Eclipse debugger to connect (rather shakily)
to Java code running inside my app.

Unfortunately, I was using Java 1.4 not 1.3, and I'm fairly sure that there
were some changes in this area -- if only in reliability.  Still, you might be
able to switch to a 1.4 JVM to do your debugging it you can't get it to work
under 1.3 (I don't /think/ it would affect Matlab too much).

Anyway to get it to work with Eclipse 2.x (which wasn't on your list, I know,
but presumably the principles are the same with other debuggers) I had to pass
flags to the jvm.dll to:
   Give -Xdebug argument
   Ensure that the Eclipse-generated .class files were first on its classpath.
   Tell it to start listening for connections from the debugger.

So the options were:
   -Xdebug
   -Djava.class.path=<...whatever...>
   -Xrunjdwp:transport=dt_socket,address=7777,server=y,suspend=n

The last option tells the JVM to start a server listening for inbound TCP/IP
connections from the debugger on port 7777 (arbitrarily chosen), and not to
wait for the connection to be established before starting up.

I then had to tell Eclipse to connect to a running Java application that was
listening on that port on 'localhost'.  Offhand I can't remember how to do
that, but since you aren't using Eclipse I don't suppose the details would help
you very much anyway.

It's some time since I did this.  IIRC there were some issues about
disconnecting from the debugged application -- it didn't work, or crashed the
application, or something like that, so be cautious...

HTH

   -- chris

see http://mindprod.com/debugging.html
http://mindprod.com/jgloss/debugger.html
http://mindprod.com/jgloss/trace.html

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.