Presumably the JVM uses RTLD_LOCAL to load liba.so, so its symbols are
not directly visible from libb.so.

Offhand I can think of two workarounds:

1. If you specify "-f libb.so" when you build liba.so, you tell the
  linker about a's dependency on b. The effect of this is that when
  your Java program does System.loadLibrary("a"), both liba.so and
  libb.so will be loaded automatically, and you don't need to do
  dlopen("libb.so") from liba.so. Both sets of symbols will be
  directly visible from both libraries (without the need for
  dlopen(), dlsym() etc).

2. From a.so, you could pass function pointers to an initialization
  function in b.so, and let b.so access the functions through the
  pointers.

You might also consider combining or at least refactoring the two
libraries in order to avoid the circular dependency.

/gordon