I'd never done it with Excel but with several other programs:
First you must reach "native" code in your program (writing vb, and
using a dll).
In this native code you can (all in JNI, sources of java.exe in JDK is
a good (even if a bit messy) example )
load a jvm
load your main class,
call the appropriate method
get the result
convert it and return it.

This is the hardcore way to proceed.
It seems to me that there is an activeX that launch the JVM, and maybe
enables code return (don't know its specification). But I've never
explore this way.