Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / First Aid / November 2005

Tip: Looking for answers? Try searching our database.

Class Loading from Byte Stream

Thread view: 
E11 - 07 Nov 2005 09:51 GMT
Hi,

The background is that i have JAR files stored as BLOBs in a database
table, and i need to load classes from these JAR files.

I understand that the easiest way out would be to write the JAR files
on to a file system, and then use the URLClassLoader to load the
classes, but what i would prefer to do is to load the classes directly
from the byte representation of the JAR files, without having to write
them to the file system first.

Now, i can attempt to write my own custom class loader for this
purpose, but there are a few things that i could do with advice on.

1. Is there already an open-source class loader out there that achieves
this purpose? (Well, obviously if there is, i won't need to re-invent
the wheel)

2. i would imagine that doing it this way is better for performance as
compared to going through the file system (and i feel its cleaner too
as it doesn't leave the JAR files around), but would there be a
substantial impact on memory? I would imagine that it needs to hold all
those bytes in memory, so does it mean that if i have a, say, 10 MB JAR
file, doing it this way would incur a 10 MB cost on memory?

TIA and Regards,
Edwin
Thomas Schodt - 07 Nov 2005 10:17 GMT
> 1. Is there already an open-source class loader out there that achieves
> this purpose? (Well, obviously if there is, i won't need to re-invent
> the wheel)

AFAICT, when you use a database, you don't store the jar, you store the
individual class files ("compilation units").

<http://java.sun.com/docs/books/jls/third_edition/html/packages.html#7.2.2>
E11 - 07 Nov 2005 10:42 GMT
> AFAICT, when you use a database, you don't store the jar, you store the
> individual class files ("compilation units").
>
> <http://java.sun.com/docs/books/jls/third_edition/html/packages.html#7.2.2>

Hmm, i have read that one, but i think my problem is of a slightly
different nature. i.e. we can take it that the JAR file is already in
the database table as a BLOB, and i have no choice over that, so i have
to either load the classes from the JAR byte stream, or write the byte
stream to a JAR file, then load the classes from there.

Regards,
Edwin
Roedy Green - 07 Nov 2005 10:51 GMT
>Hmm, i have read that one, but i think my problem is of a slightly
>different nature. i.e. we can take it that the JAR file is already in
>the database table as a BLOB, and i have no choice over that, so i have
>to either load the classes from the JAR byte stream, or write the byte
>stream to a JAR file, then load the classes from there.

That makes no sense  unless you plan to preemptively load the entire
jar at once.  Open the jar and create BLOBs for each class.

Further, how to you know which classes are in which jar?  You still
need an index by individual class files do you not?

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 07 Nov 2005 11:22 GMT
On Mon, 07 Nov 2005 10:51:02 GMT, Roedy Green
<my_email_is_posted_on_my_website@munged.invalid> wrote, quoted or
indirectly quoted someone who said :

>That makes no sense  unless you plan to preemptively load the entire
>jar at once.  Open the jar and create BLOBs for each class.
>
>Further, how to you know which classes are in which jar?  You still
>need an index by individual class files do you not?

If you want to deal with adding and replacing on a jar level rather
than a class level here is what you can do:

In your jar record assign it a sequence number as well as a unique
name..   Then when you open the jar and pull out the classes inside
you can create a blob for each class, indexed by packagename / class
and jar sequence number.

Now when you get a new jar, you can kill all the old class files with
a delete on jar number.

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 07 Nov 2005 11:45 GMT
On Mon, 07 Nov 2005 11:22:22 GMT, Roedy Green
<my_email_is_posted_on_my_website@munged.invalid> wrote, quoted or
indirectly quoted someone who said :

>Now when you get a new jar, you can kill all the old class files with
>a delete on jar number.

If you track the timestamp of each class, all you need do in update
the jar number when a class has not really changed.  That way the same
BLOB will do. Saves some "wear and tear" on the database.

A BLOB is a messy thing. Because it is massively variable length, it
will typically be allocated from some pool of space separate from the
other data for the record. That pool will need to be defragged more
frequently if you keep needlessly creating and deleting blobs.
Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

E11 - 07 Nov 2005 12:02 GMT
> That makes no sense  unless you plan to preemptively load the entire
> jar at once.  Open the jar and create BLOBs for each class.
>
> Further, how to you know which classes are in which jar?  You still
> need an index by individual class files do you not?

Well, yes, i do intend to load the entire JAR at once, and like i said,
the JAR is already in the database table as one BLOB, and i have no
choice to store each class file as an individual BLOB (well unless i
read the JAR BLOB, unJAR it into individual class byte streams, store
each class byte stream in its own BLOB, then read from there again but
that's contrived.)

Regards,
Edwin
Roedy Green - 07 Nov 2005 12:19 GMT
>Well, yes, i do intend to load the entire JAR at once, and like i said,
>the JAR is already in the database table as one BLOB, and i have no
>choice to store each class file as an individual BLOB (well unless i
>read the JAR BLOB, unJAR it into individual class byte streams, store
>each class byte stream in its own BLOB, then read from there again but
>that's contrived.)

It makes perfect sense.  You unpack only once, then load classes
individually many times as needed with a simple class loader that does
not need to cache an entire jar or understand jar structure.

You are making life slightly more difficult for yourself since there
is no hook to load a jar in Java, just individual classes.

What you will need to do is instantiate a classLoader that takes a JAR
name as a parameter. It wakes up, does a query, and gets an in-ram
copy of the jar and drops the SQL connection.

Now your application code will have to call a custom method of the
classloader to divulge a list of the classes inside.  You now then do
a classForName on each one using that custom class loader.  When you
have all the classes loaded, you then have to call a custom method to
get the classloader to discard the jar image. Don't discard the link
to the classloader until you are ready to discard all the loaded
classes too.

Now you can get at those classes via interfaces, not their normal
class names because they are not the same classes as if they were
loaded with the usual class loader.

That is a result of using a classloader, not using a jar BLOB..

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 07 Nov 2005 12:31 GMT
On Mon, 07 Nov 2005 12:19:26 GMT, Roedy Green
<my_email_is_posted_on_my_website@munged.invalid> wrote, quoted or
indirectly quoted someone who said :

>Now you can get at those classes via interfaces, not their normal
>class names because they are not the same classes as if they were
>loaded with the usual class loader.

You say, you can get at instances of these classes via interfaces. You
can get at the classes themselves via Class objects.  Normally you
want to do everything via an interface.
Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

E11 - 07 Nov 2005 13:02 GMT
> It makes perfect sense.  You unpack only once, then load classes
> individually many times as needed with a simple class loader that does
> not need to cache an entire jar or understand jar structure.

Well, actually we seem to have very much the same idea save for some
differences in vocabulary, so please allow me to clarify.

When i said BLOB, i'm referring to Binary Large OBject, the thing in
the database table, once its taken out of the database table, its just
a stream of bytes. (Of course, some API needed to get the stream of
bytes from the BLOB.)

This is what i have:
A JAR file in the form of a BLOB on the database table.

What i need:
To load classes from the JAR file.

And these are the steps i had in mind:
1. Get the stream of bytes representing the JAR file from the BLOB in
the database table.
2. Load the stream of bytes representing the JAR file and unpack them,
so i can load individual classes as and when needed.
3. What i DID miss out was the unloading of the byte stream from
memory, thanks for that, i think that would solve the problem of the
huge memory footprint.

My question then, is whether there are already source code out there
for doing step 2.

Perhaps i should have decomposed my problem and phrased it as such:
Given a byte stream representing a JAR file, how best to load classes
from it without writing it to the file system first?

Thanks and Regards,
Edwin
Roedy Green - 07 Nov 2005 13:31 GMT
>2. Load the stream of bytes representing the JAR file and unpack them,
>so i can load individual classes as and when needed.

This will be a bit clumsy.  You can't use the ZipFile class with the
index since it requires an actual flat file. Instead you will have to
take your byte[] and convert that to a ByteArrayInputStream see
http://mindprod.com/applets/fileio.html for how.

Use ZipInputStream to create the list of classes you have on tap.  You
might consider splitting package and class name and interning the
package name to save space if your jars are enormous.

Then sequentially read the whole file searching for the class of
interest with ZipInputStream. See http://mindprod.com/jgloss/zip.html

Since everything is in RAM this won't be that painful.

Make sure to nullify everything you can after the classes are loaded,
e.g. your index of classes,  otherwise they will hang in there till
the end.

I think you will find it much easier to take your jar apart and create
separate blobs for each class, then use a relatively mindless
classloader that just does a query for the class and loads it. Granted
the jar-at-a-pop approach will be faster.

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 07 Nov 2005 13:35 GMT
>Perhaps i should have decomposed my problem and phrased it as such:
>Given a byte stream representing a JAR file, how best to load classes
>from it without writing it to the file system first?

If you wrote it to disk, you could use ZipFile. But with the
ByteArrayInputStream/ZipInputStream technique I explained in my
previous post, you can still do it.  It just requires a tedious linear
search through the body index (as opposed to the summary index at the
end of the zip) to find each class.

If this task becomes too daunting, I could write it for you for a fee.
Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

E11 - 08 Nov 2005 03:10 GMT
I have come up with this (below). Have tested it and it seems to work,
but i have two concerns:

1. Is there anything that i might have done wrongly? i.e. anything that
could break it in future?

2. What if one of the classes in the "jar bytes" refer to a properties
file that is jarred together? How should this be handled? Looking at
the API for ClassLoader, it seems that i should override findResource,
but that method returns a URL, so i'm not sure what to return if i do
override it.

import EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap;

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Map;
import java.util.jar.JarEntry;
import java.util.jar.JarInputStream;

public class JarBytesClassLoader extends ClassLoader
{
    private static JarBytesClassLoader jarBytesClassLoader = new
JarBytesClassLoader();

    private Map classesMap; // Fully Qualified Class Name (String) to Byte
Array (byte[])

    private JarBytesClassLoader()
    {
        this.classesMap = new ConcurrentReaderHashMap();
    }

    public static JarBytesClassLoader getInstance()
    {
        return jarBytesClassLoader;
    }

    public void addJarBytes(byte[] jarBytes)
    {
        ByteArrayInputStream byteArrayInputStream = null;
        JarInputStream jarInputStream = null;
        BufferedInputStream bufferedInputStream = null;
        try
        {
            byteArrayInputStream = new ByteArrayInputStream(jarBytes);
            jarInputStream = new JarInputStream(byteArrayInputStream);
            bufferedInputStream = new BufferedInputStream(jarInputStream);

            JarEntry jarEntry = null;
            while ((jarEntry = jarInputStream.getNextJarEntry()) != null)
            {
                if (!jarEntry.isDirectory())
                {
                    String name = jarEntry.getName();
                    if ((name.toLowerCase().endsWith(".class")))
                    {
                        String className = name.replaceAll("/", ".").substring(0,
(name.length() - 6));
                        // System.out.println(className);

                        ByteArrayOutputStream byteArrayOutputStream = null;
                        BufferedOutputStream bufferedOutputStream = null;
                        try
                        {
                            byteArrayOutputStream = new ByteArrayOutputStream();
                            bufferedOutputStream = new
BufferedOutputStream(byteArrayOutputStream);

                            int i = -1;
                            while ((i = bufferedInputStream.read()) != -1)
                            {
                                bufferedOutputStream.write(i);
                            }

                            bufferedOutputStream.flush();

                            byte[] bytes = byteArrayOutputStream.toByteArray();
                            this.classesMap.put(className, bytes);
                        }
                        finally
                        {
                            if (byteArrayOutputStream != null)
                            {
                                try
                                {
                                    byteArrayOutputStream.close();
                                }
                                catch (IOException ex)
                                {
                                    // Ignore IOException When Closing
                                }
                            }

                            if (bufferedOutputStream != null)
                            {
                                try
                                {
                                    bufferedOutputStream.close();
                                }
                                catch (IOException ex)
                                {
                                    // Ignore IOException When Closing
                                }
                            }
                        }
                    }
                }

                jarInputStream.closeEntry();
            }
        }
        catch (IOException ex)
        {
            // TODO
            ex.printStackTrace();
        }
        finally
        {
            if (byteArrayInputStream != null)
            {
                try
                {
                    byteArrayInputStream.close();
                }
                catch (IOException ex)
                {
                    // Ignore IOException When Closing
                }
            }

            if (jarInputStream != null)
            {
                try
                {
                    jarInputStream.close();
                }
                catch (IOException ex)
                {
                    // Ignore IOException When Closing
                }
            }

            if (bufferedInputStream != null)
            {
                try
                {
                    bufferedInputStream.close();
                }
                catch (IOException ex)
                {
                    // Ignore IOException When Closing
                }
            }
        }
    }

    public static void reset()
    {
        jarBytesClassLoader.classesMap.clear();
        jarBytesClassLoader = new JarBytesClassLoader();
    }

    protected Class findClass(String name)
    {
        byte[] bytes = loadClassData(name);
        return (defineClass(name, bytes, 0, bytes.length));
    }

    private byte[] loadClassData(String name)
    {
        return ((byte[]) this.classesMap.get(name));
    }
}

TIA and Regards,
Edwin
Roedy Green - 08 Nov 2005 04:52 GMT
>int i = -1;
>                            while ((i = bufferedInputStream.read()) != -1)
>                            {
>                                bufferedOutputStream.write(i);
>                            }

You might as well read it directly into a buffer of the correct length
in one I/O if you know ahead of time how big to make it.  I vaguely
recall though that Java gives you 0 lengths for members since the file
was created in one pass with ZipOutputStream, it could not go back and
patch the leading lengths.

If this code turns out to be a bottleneck, you can read the class
contents a chunk at a time rather than a byte at a time. See
http://mindprod.com/products1.html#FILETRANSFER
for how.

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 08 Nov 2005 04:54 GMT
>        byteArrayOutputStream = new ByteArrayOutputStream();
>                   
if you don't make at least a wild stab at a size it will start small
and grow, like an ArrayList, taking time out to copy to the next
bigger buffer at each stage, littering the heap with large discarded
buffers.
Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 08 Nov 2005 04:57 GMT
>private byte[] loadClassData(String name)
>    {
>        return ((byte[]) this.classesMap.get(name));
>    }
this is a misnamed method. It does not load the class. It just get
the bytes for it.
Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 08 Nov 2005 04:58 GMT
>while ((jarEntry = jarInputStream.getNextJarEntry()) != null)
>            {
>           

Your method is superior to the one I outlined.  You have to parse the
jar file only once, not once per class.
Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 08 Nov 2005 04:59 GMT
>    public void addJarBytes(byte[] jarBytes)

make sure as soon as you call this method to effectively set jarBytes
= null;
Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Roedy Green - 07 Nov 2005 10:42 GMT
>The background is that i have JAR files stored as BLOBs in a database
>table, and i need to load classes from these JAR files.

I would think you want to put each class in its own BLOB. Then you can
use the indexing features of SQL to find just the class you need.
Otherwise you have to get the ENTIRE jar, and poke around in it just
to load one class -- very inefficient.

You would write a custom class loader to fetch the bytes of the class
from SQL and hand them off to java.lang.ClassLoader.defineClass  See
http://mindprod.com/jgloss/classloader.html

It is much simpler than you might imagine.

You might add a connect and disconnect method to your classloader so
you could load a batch of classes without building a fresh JDBC
connection for each class load.
Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.



Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.