Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / May 2006

Tip: Looking for answers? Try searching our database.

memory consumption in JDOM

Thread view: 
vanandh22@gmail.com - 12 May 2006 13:03 GMT
Hi All
I am new to JDOM  and I am currently coding a program to build xml
files using JDOM.
The size of the xml file which i am goin to build will be in the order
of a few GB's( 1 to 5 GB).
Will I be able to build such huge documents with JDOM??
 Also if there are any resources on jdom benchmarking please let me
know the links.

Thanks in Advance for your help
Vijay Anandh
Vijay Anandh - 13 May 2006 18:41 GMT
Any updates on this issue....,
I am eagerly waiting for the answers.
Chris Uppal - 14 May 2006 10:00 GMT
>  I am new to JDOM  and I am currently coding a program to build xml
> files using JDOM.
>  The size of the xml file which i am goin to build will be in the order
> of a few GB's( 1 to 5 GB).

I'm not sure what the typical overhead of a JDOM structure is (how much RAM it
takes compared to how much disk-space the equivalent XML would take up), and
it's application-dependent anyway.  Assume, for the sake of argument, that it's
3:1.

Under that assumption you'll need around 3 to 15 GB of addressable RAM just to
hold the JDOM structure.  So you will have to be running on a 64-bit JVM before
you can even consider this.

Then there's the space required for the JVM itself.  And possibly the space
required for the data from which you build your JDOM tree.  I would hope that
you don't try to convert the entire tree into a String before writing it out to
file (or to the network), but maybe you would have to do so.  So, when you are
considering how much real RAM you need, you have to think in terms of at least
1 GB more than the above estimate, and maybe even (in the worst case) more than
double the estimate.

Only you know whether your application can justify using those kinds of
resources.

Personally, I would think /very/ hard about why I cannot just build the XML
incrementally (writing it out to file in one pass without ever holding all the
data in memory at once).  With luck, some thought would show that it is
possible after all.

   -- chris
Domagoj Klepac - 15 May 2006 10:39 GMT
>I'm not sure what the typical overhead of a JDOM structure is (how much RAM it
>takes compared to how much disk-space the equivalent XML would take up), and
>it's application-dependent anyway.  Assume, for the sake of argument, that it's
>3:1.

Yes, I'd say that's about right. On the one hand, Java stores Strings
as Unicode, so one character takes 2 bytes - one byte more than on
disk; on the other hand, if you have:

<longxmlelementname>foo</longxmlelementname>

You have to store only two Strings in memory:

longxmlelementname foo

And of course, there's JDOM element overhead. There's the good article
on Java object overhead and memory consumption:

http://www.javaworld.com/javaworld/javatips/jw-javatip130.html

>Under that assumption you'll need around 3 to 15 GB of addressable RAM just to
>hold the JDOM structure.  So you will have to be running on a 64-bit JVM before
>you can even consider this.

It seems that the speed limit is the one you hit before the memory
limit. If it takes a hours to fill even 500 MB of memory with XML
data...

>Personally, I would think /very/ hard about why I cannot just build the XML
>incrementally (writing it out to file in one pass without ever holding all the
>data in memory at once).  With luck, some thought would show that it is
>possible after all.

Yes, some kind of SAX parser which doesn't load XML to memory would
probably be more appropriate.

               Domchi

Signature

Ouroboros ltd. - http://www.ouroboros.hr 
Antispam: to reply, remove extra monkey from reply-to address.

Domagoj Klepac - 15 May 2006 10:21 GMT
> I am new to JDOM  and I am currently coding a program to build xml
>files using JDOM.
[quoted text clipped - 3 lines]
>  Also if there are any resources on jdom benchmarking please let me
>know the links.

At a time, I found a few JDOM benchmarks, mostly bechmarking speed.
JDOM scored pretty well compared to purely SAX tools, it is one of the
fastest.

As for the maximum size of an XML file... why not test it yourself?

I've just run this simple test case:

import org.jdom.Document;
import org.jdom.Element;

public class JDOM {
 // Timeout set to 60 seconds
 private static long timeout = 60 * 1000;
   
 public static void main(String[] args) {
   Element element = new Element("root");
   Document document = new Document(element);
   long i = 1;
   long startTime = System.currentTimeMillis();
       
   try {
     for (;; i++) {
       Element currElement = new Element("element" +
                                                  String.valueOf(i));
       element.addContent(currElement);
       element = currElement;
       if (System.currentTimeMillis() - startTime > timeout){
         break;
       }
     }
   } catch (Exception ex) {
     System.out.println("Exception: " +  ex.getMessage());
   }

   System.out.println("Number of created elements: " + i);
 }
}

It was able to create 28885 elements in 60 seconds. I left it running
for a while without a timeout, to see if it would throw
OutOfMemoryException, but it seems that the memory overhead for
element is pretty low, so it would take forever to drain the heap.

               Domchi

Signature

Ouroboros ltd. - http://www.ouroboros.hr 
Antispam: to reply, remove extra monkey from reply-to address.

Dale King - 15 May 2006 18:53 GMT
> Hi All
>  I am new to JDOM  and I am currently coding a program to build xml
> files using JDOM.

A better alternative to JDOM that you might want to consider is XOM
(www.xom.nu) which is much cleaner and easier to use.
Signature

 Dale King



Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.