Java Forum / First Aid / November 2005
Class Loading from Byte Stream
E11 - 07 Nov 2005 09:51 GMT Hi,
The background is that i have JAR files stored as BLOBs in a database table, and i need to load classes from these JAR files.
I understand that the easiest way out would be to write the JAR files on to a file system, and then use the URLClassLoader to load the classes, but what i would prefer to do is to load the classes directly from the byte representation of the JAR files, without having to write them to the file system first.
Now, i can attempt to write my own custom class loader for this purpose, but there are a few things that i could do with advice on.
1. Is there already an open-source class loader out there that achieves this purpose? (Well, obviously if there is, i won't need to re-invent the wheel)
2. i would imagine that doing it this way is better for performance as compared to going through the file system (and i feel its cleaner too as it doesn't leave the JAR files around), but would there be a substantial impact on memory? I would imagine that it needs to hold all those bytes in memory, so does it mean that if i have a, say, 10 MB JAR file, doing it this way would incur a 10 MB cost on memory?
TIA and Regards, Edwin
Thomas Schodt - 07 Nov 2005 10:17 GMT > 1. Is there already an open-source class loader out there that achieves > this purpose? (Well, obviously if there is, i won't need to re-invent > the wheel) AFAICT, when you use a database, you don't store the jar, you store the individual class files ("compilation units").
<http://java.sun.com/docs/books/jls/third_edition/html/packages.html#7.2.2>
E11 - 07 Nov 2005 10:42 GMT > AFAICT, when you use a database, you don't store the jar, you store the > individual class files ("compilation units"). > > <http://java.sun.com/docs/books/jls/third_edition/html/packages.html#7.2.2> Hmm, i have read that one, but i think my problem is of a slightly different nature. i.e. we can take it that the JAR file is already in the database table as a BLOB, and i have no choice over that, so i have to either load the classes from the JAR byte stream, or write the byte stream to a JAR file, then load the classes from there.
Regards, Edwin
Roedy Green - 07 Nov 2005 10:51 GMT >Hmm, i have read that one, but i think my problem is of a slightly >different nature. i.e. we can take it that the JAR file is already in >the database table as a BLOB, and i have no choice over that, so i have >to either load the classes from the JAR byte stream, or write the byte >stream to a JAR file, then load the classes from there. That makes no sense unless you plan to preemptively load the entire jar at once. Open the jar and create BLOBs for each class.
Further, how to you know which classes are in which jar? You still need an index by individual class files do you not?
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 07 Nov 2005 11:22 GMT On Mon, 07 Nov 2005 10:51:02 GMT, Roedy Green <my_email_is_posted_on_my_website@munged.invalid> wrote, quoted or indirectly quoted someone who said :
>That makes no sense unless you plan to preemptively load the entire >jar at once. Open the jar and create BLOBs for each class. > >Further, how to you know which classes are in which jar? You still >need an index by individual class files do you not? If you want to deal with adding and replacing on a jar level rather than a class level here is what you can do:
In your jar record assign it a sequence number as well as a unique name.. Then when you open the jar and pull out the classes inside you can create a blob for each class, indexed by packagename / class and jar sequence number.
Now when you get a new jar, you can kill all the old class files with a delete on jar number.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 07 Nov 2005 11:45 GMT On Mon, 07 Nov 2005 11:22:22 GMT, Roedy Green <my_email_is_posted_on_my_website@munged.invalid> wrote, quoted or indirectly quoted someone who said :
>Now when you get a new jar, you can kill all the old class files with >a delete on jar number. If you track the timestamp of each class, all you need do in update the jar number when a class has not really changed. That way the same BLOB will do. Saves some "wear and tear" on the database.
A BLOB is a messy thing. Because it is massively variable length, it will typically be allocated from some pool of space separate from the other data for the record. That pool will need to be defragged more frequently if you keep needlessly creating and deleting blobs.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
E11 - 07 Nov 2005 12:02 GMT > That makes no sense unless you plan to preemptively load the entire > jar at once. Open the jar and create BLOBs for each class. > > Further, how to you know which classes are in which jar? You still > need an index by individual class files do you not? Well, yes, i do intend to load the entire JAR at once, and like i said, the JAR is already in the database table as one BLOB, and i have no choice to store each class file as an individual BLOB (well unless i read the JAR BLOB, unJAR it into individual class byte streams, store each class byte stream in its own BLOB, then read from there again but that's contrived.)
Regards, Edwin
Roedy Green - 07 Nov 2005 12:19 GMT >Well, yes, i do intend to load the entire JAR at once, and like i said, >the JAR is already in the database table as one BLOB, and i have no >choice to store each class file as an individual BLOB (well unless i >read the JAR BLOB, unJAR it into individual class byte streams, store >each class byte stream in its own BLOB, then read from there again but >that's contrived.) It makes perfect sense. You unpack only once, then load classes individually many times as needed with a simple class loader that does not need to cache an entire jar or understand jar structure.
You are making life slightly more difficult for yourself since there is no hook to load a jar in Java, just individual classes.
What you will need to do is instantiate a classLoader that takes a JAR name as a parameter. It wakes up, does a query, and gets an in-ram copy of the jar and drops the SQL connection.
Now your application code will have to call a custom method of the classloader to divulge a list of the classes inside. You now then do a classForName on each one using that custom class loader. When you have all the classes loaded, you then have to call a custom method to get the classloader to discard the jar image. Don't discard the link to the classloader until you are ready to discard all the loaded classes too.
Now you can get at those classes via interfaces, not their normal class names because they are not the same classes as if they were loaded with the usual class loader.
That is a result of using a classloader, not using a jar BLOB..
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 07 Nov 2005 12:31 GMT On Mon, 07 Nov 2005 12:19:26 GMT, Roedy Green <my_email_is_posted_on_my_website@munged.invalid> wrote, quoted or indirectly quoted someone who said :
>Now you can get at those classes via interfaces, not their normal >class names because they are not the same classes as if they were >loaded with the usual class loader. You say, you can get at instances of these classes via interfaces. You can get at the classes themselves via Class objects. Normally you want to do everything via an interface.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
E11 - 07 Nov 2005 13:02 GMT > It makes perfect sense. You unpack only once, then load classes > individually many times as needed with a simple class loader that does > not need to cache an entire jar or understand jar structure. Well, actually we seem to have very much the same idea save for some differences in vocabulary, so please allow me to clarify.
When i said BLOB, i'm referring to Binary Large OBject, the thing in the database table, once its taken out of the database table, its just a stream of bytes. (Of course, some API needed to get the stream of bytes from the BLOB.)
This is what i have: A JAR file in the form of a BLOB on the database table.
What i need: To load classes from the JAR file.
And these are the steps i had in mind: 1. Get the stream of bytes representing the JAR file from the BLOB in the database table. 2. Load the stream of bytes representing the JAR file and unpack them, so i can load individual classes as and when needed. 3. What i DID miss out was the unloading of the byte stream from memory, thanks for that, i think that would solve the problem of the huge memory footprint.
My question then, is whether there are already source code out there for doing step 2.
Perhaps i should have decomposed my problem and phrased it as such: Given a byte stream representing a JAR file, how best to load classes from it without writing it to the file system first?
Thanks and Regards, Edwin
Roedy Green - 07 Nov 2005 13:31 GMT >2. Load the stream of bytes representing the JAR file and unpack them, >so i can load individual classes as and when needed. This will be a bit clumsy. You can't use the ZipFile class with the index since it requires an actual flat file. Instead you will have to take your byte[] and convert that to a ByteArrayInputStream see http://mindprod.com/applets/fileio.html for how.
Use ZipInputStream to create the list of classes you have on tap. You might consider splitting package and class name and interning the package name to save space if your jars are enormous.
Then sequentially read the whole file searching for the class of interest with ZipInputStream. See http://mindprod.com/jgloss/zip.html
Since everything is in RAM this won't be that painful.
Make sure to nullify everything you can after the classes are loaded, e.g. your index of classes, otherwise they will hang in there till the end.
I think you will find it much easier to take your jar apart and create separate blobs for each class, then use a relatively mindless classloader that just does a query for the class and loads it. Granted the jar-at-a-pop approach will be faster.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 07 Nov 2005 13:35 GMT >Perhaps i should have decomposed my problem and phrased it as such: >Given a byte stream representing a JAR file, how best to load classes >from it without writing it to the file system first? If you wrote it to disk, you could use ZipFile. But with the ByteArrayInputStream/ZipInputStream technique I explained in my previous post, you can still do it. It just requires a tedious linear search through the body index (as opposed to the summary index at the end of the zip) to find each class.
If this task becomes too daunting, I could write it for you for a fee.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
E11 - 08 Nov 2005 03:10 GMT I have come up with this (below). Have tested it and it seems to work, but i have two concerns:
1. Is there anything that i might have done wrongly? i.e. anything that could break it in future?
2. What if one of the classes in the "jar bytes" refer to a properties file that is jarred together? How should this be handled? Looking at the API for ClassLoader, it seems that i should override findResource, but that method returns a URL, so i'm not sure what to return if i do override it.
import EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap;
import java.io.BufferedInputStream; import java.io.BufferedOutputStream; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.FileInputStream; import java.io.IOException; import java.util.Map; import java.util.jar.JarEntry; import java.util.jar.JarInputStream;
public class JarBytesClassLoader extends ClassLoader { private static JarBytesClassLoader jarBytesClassLoader = new JarBytesClassLoader();
private Map classesMap; // Fully Qualified Class Name (String) to Byte Array (byte[])
private JarBytesClassLoader() { this.classesMap = new ConcurrentReaderHashMap(); }
public static JarBytesClassLoader getInstance() { return jarBytesClassLoader; }
public void addJarBytes(byte[] jarBytes) { ByteArrayInputStream byteArrayInputStream = null; JarInputStream jarInputStream = null; BufferedInputStream bufferedInputStream = null; try { byteArrayInputStream = new ByteArrayInputStream(jarBytes); jarInputStream = new JarInputStream(byteArrayInputStream); bufferedInputStream = new BufferedInputStream(jarInputStream);
JarEntry jarEntry = null; while ((jarEntry = jarInputStream.getNextJarEntry()) != null) { if (!jarEntry.isDirectory()) { String name = jarEntry.getName(); if ((name.toLowerCase().endsWith(".class"))) { String className = name.replaceAll("/", ".").substring(0, (name.length() - 6)); // System.out.println(className);
ByteArrayOutputStream byteArrayOutputStream = null; BufferedOutputStream bufferedOutputStream = null; try { byteArrayOutputStream = new ByteArrayOutputStream(); bufferedOutputStream = new BufferedOutputStream(byteArrayOutputStream);
int i = -1; while ((i = bufferedInputStream.read()) != -1) { bufferedOutputStream.write(i); }
bufferedOutputStream.flush();
byte[] bytes = byteArrayOutputStream.toByteArray(); this.classesMap.put(className, bytes); } finally { if (byteArrayOutputStream != null) { try { byteArrayOutputStream.close(); } catch (IOException ex) { // Ignore IOException When Closing } }
if (bufferedOutputStream != null) { try { bufferedOutputStream.close(); } catch (IOException ex) { // Ignore IOException When Closing } } } } }
jarInputStream.closeEntry(); } } catch (IOException ex) { // TODO ex.printStackTrace(); } finally { if (byteArrayInputStream != null) { try { byteArrayInputStream.close(); } catch (IOException ex) { // Ignore IOException When Closing } }
if (jarInputStream != null) { try { jarInputStream.close(); } catch (IOException ex) { // Ignore IOException When Closing } }
if (bufferedInputStream != null) { try { bufferedInputStream.close(); } catch (IOException ex) { // Ignore IOException When Closing } } } }
public static void reset() { jarBytesClassLoader.classesMap.clear(); jarBytesClassLoader = new JarBytesClassLoader(); }
protected Class findClass(String name) { byte[] bytes = loadClassData(name); return (defineClass(name, bytes, 0, bytes.length)); }
private byte[] loadClassData(String name) { return ((byte[]) this.classesMap.get(name)); } }
TIA and Regards, Edwin
Roedy Green - 08 Nov 2005 04:52 GMT >int i = -1; > while ((i = bufferedInputStream.read()) != -1) > { > bufferedOutputStream.write(i); > } You might as well read it directly into a buffer of the correct length in one I/O if you know ahead of time how big to make it. I vaguely recall though that Java gives you 0 lengths for members since the file was created in one pass with ZipOutputStream, it could not go back and patch the leading lengths.
If this code turns out to be a bottleneck, you can read the class contents a chunk at a time rather than a byte at a time. See http://mindprod.com/products1.html#FILETRANSFER for how.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 08 Nov 2005 04:54 GMT > byteArrayOutputStream = new ByteArrayOutputStream(); > if you don't make at least a wild stab at a size it will start small and grow, like an ArrayList, taking time out to copy to the next bigger buffer at each stage, littering the heap with large discarded buffers.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 08 Nov 2005 04:57 GMT >private byte[] loadClassData(String name) > { > return ((byte[]) this.classesMap.get(name)); > } this is a misnamed method. It does not load the class. It just get the bytes for it.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 08 Nov 2005 04:58 GMT >while ((jarEntry = jarInputStream.getNextJarEntry()) != null) > { > Your method is superior to the one I outlined. You have to parse the jar file only once, not once per class.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 08 Nov 2005 04:59 GMT > public void addJarBytes(byte[] jarBytes) make sure as soon as you call this method to effectively set jarBytes = null;
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Roedy Green - 07 Nov 2005 10:42 GMT >The background is that i have JAR files stored as BLOBs in a database >table, and i need to load classes from these JAR files. I would think you want to put each class in its own BLOB. Then you can use the indexing features of SQL to find just the class you need. Otherwise you have to get the ENTIRE jar, and poke around in it just to load one class -- very inefficient.
You would write a custom class loader to fetch the bytes of the class from SQL and hand them off to java.lang.ClassLoader.defineClass See http://mindprod.com/jgloss/classloader.html
It is much simpler than you might imagine.
You might add a connect and disconnect method to your classloader so you could load a batch of classes without building a fresh JDBC connection for each class load.
 Signature Canadian Mind Products, Roedy Green. http://mindprod.com Java custom programming, consulting and coaching.
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|