 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / General / June 2007Memory Profiler for Heap Analysis
Hi, I have an application (xml-editor) that uses up an unreasonable amount of java heap space. I have tried profiling with tptp (eclipse) and jmap/jhat, and they do give me memory histograms and instance counts, but what I REALLY want is to see the total size of a classes instances INCLUDING the referenced objects. Is there a profiler that will traverse the heap and find the "recursive" size of objects? Because knowing that char-Arrays use up most memory is not exactly helpful. Plese help, I've searched the web and found nothing; but have been assured that something like this exists. TIA, Moritz > I have an application (xml-editor) that uses up an unreasonable amount > of java heap space. [quoted text clipped - 9 lines] > Plese help, I've searched the web and found nothing; but have been > assured that something like this exists. I don't think something like that exists. Assuming something like this exists, how does it properly calculate sizes? How many levels does it follow when counting? How do you count objects that are referenced by multiple other instances etc. Having said that I was pretty satisfied with OptimizeIT which also does not do recursive analysis but lets you easily find places in code where all those objects were allocated and you could walk the object graph. I haven't used it in a while so maybe today there are other (free?) tools around that server similar purposes. Kind regards robert Hi Robert, > I don't think something like that exists. Assuming something like this > exists, how does it properly calculate sizes? How many levels does it > follow when counting? Until it hits a cycle? > How do you count objects that are referenced by multiple other > instances etc. Add the objects size to both instaces...? Well, I was sceptical too as to wheather this was even possible, but when I asked for this feature in the TPTP (eclipse profiler) group, they replied > This is a valid and useful use-case for memory profiling. While > technically possible (both in theory and in practice), it is not > currently > in the plan for TPTP. So I thought this might be implemented, if not in a free, then in a commercial tool. Of course it's not a trivial (or probably fast for that matter) task, but I guess you could somehow work around the problems. I don't want a 100% accurate size count, just an overview over which classes use up most memory. I find it kind of hard to see with the standard "heap historgram" but maybe I'm missing something. Does anyone have any other suggestions? As stated I have an application (edior) that uses too much memory (about 21 MB for each MB file size loaded) and want to see where all of this goes (DOM, Graphical representation, etc..) Moritz > Hi Robert, > [quoted text clipped - 3 lines] > > Until it hits a cycle? If it does not? >> How do you count objects that are referenced by multiple other >> instances etc. > > Add the objects size to both instaces...? Remember that n is not restricted to 2, could be much higher. > Well, I was sceptical too as to wheather this was even possible, but > when I asked for this feature in the TPTP (eclipse profiler) group, they [quoted text clipped - 10 lines] > Of course it's not a trivial (or probably fast for that matter) task, > but I guess you could somehow work around the problems. I assume the problems are solvable more on a theoretical level. I don't see this problem practically solvable. But maybe I'm missing something - the TPTP folks surely have a better understanding of the matter. > I don't want a 100% accurate size count, just an overview over which > classes use up most memory. I find it kind of hard to see with the [quoted text clipped - 4 lines] > loaded) and want to see where all of this goes (DOM, Graphical > representation, etc..) DOM is a good candidate. If I were you, I'd create an application object model, throw away the DOM parsing and create a SAX parser which directly creates the model. Just my 0.02 EUR... Kind regards robert >>> this exists, how does it properly calculate sizes? How many levels >>> does it follow when counting? >> Until it hits a cycle? > > If it does not? Well, then all the better: it's a proper tree and at some point you'll have traversed it completely. >>> How do you count objects that are referenced by multiple other >>> instances etc. >> Add the objects size to both instaces...? > > Remember that n is not restricted to 2, could be much higher. The point you're interested in is the size of a certain object, say A. You go ahead and look at what other objects it points. Say it points to object B. Since A points to B, B's size needs to be added to A's size. It doesn't matter what other objects point to B, for it will always be referenced from A, thus, is part of A no matter what. On the other hand: I'm looking for things to REMOVE from the program. So, if you would remove A in the above scenario, B would still remain in memory, so it would be misleading. Oh well. If we continue discussing, we could as well implement it. Are you in ? ;) > DOM is a good candidate. If I were you, I'd create an application > object model, throw away the DOM parsing and create a SAX parser which > directly creates the model. Just my 0.02 EUR... I think the things we want to achieve (online validation for example) do require a DOM... It's just frustrating. I have the heap dump in front of my eyes, I can see all the classes of the DOM, still I can't figure out just how much of the memory they cost in total. Moritz >>>> this exists, how does it properly calculate sizes? How many levels >>>> does it follow when counting? [quoted text clipped - 19 lines] > So, if you would remove A in the above scenario, B would still remain in > memory, so it would be misleading. Strictly speaking, B-the-object's size is not part of A; A only holds a reference to B, not the object itself.  SignatureLew >>> Remember that n is not restricted to 2, could be much higher. >> The point you're interested in is the size of a certain object, say A. [quoted text clipped - 8 lines] > Strictly speaking, B-the-object's size is not part of A; A only holds a > reference to B, not the object itself. Of course. But then the size of A is not representative of memory usage. >>>> this exists, how does it properly calculate sizes? How many levels >>>> does it follow when counting? [quoted text clipped - 4 lines] > Well, then all the better: it's a proper tree and at some point you'll > have traversed it completely. This still does not address the issue of aliasing properly. >>>> How do you count objects that are referenced by multiple other >>>> instances etc. [quoted text clipped - 3 lines] > > The point you're interested in is the size of a certain object, say A. The point I am trying to make is that this concept does not exist with regard to /referenced/ objects. The only size you can reasonably well detect is the size of the object itself. > You go ahead and look at what other objects it points. Say it points to > object B. Since A points to B, B's size needs to be added to A's size. [quoted text clipped - 6 lines] > Oh well. If we continue discussing, we could as well implement it. Are > you in ? ;) LOL - no. >> DOM is a good candidate. If I were you, I'd create an application >> object model, throw away the DOM parsing and create a SAX parser which >> directly creates the model. Just my 0.02 EUR... > > I think the things we want to achieve (online validation for example) do > require a DOM... If it is a general XML editor then yes, maybe. If you are editing a specific XML format, then no. > It's just frustrating. I have the heap dump in front of my eyes, I can > see all the classes of the DOM, still I can't figure out just how much > of the memory they cost in total. Did you try the memory profiler from Eclipse's TPTP? How did it go? Kind regards robert > The point I am trying to make is that this concept does not exist with > regard to /referenced/ objects. The only size you can reasonably well > detect is the size of the object itself. Point taken. That was never the question, but more: -given you know what you are doing -are able to tune some parameters -is there a reasonable approximation I thought (and still think) there must be a way >>> DOM is a good candidate. If I were you, I'd create an application >>> object model, throw away the DOM parsing and create a SAX parser which [quoted text clipped - 4 lines] > If it is a general XML editor then yes, maybe. If you are editing a > specific XML format, then no. General XML editor. I have to look into it some more, but 20/1 Memory Usage/Document Size seems a little much. Maybe that's what they mean by "Java is memory intensive" >> It's just frustrating. I have the heap dump in front of my eyes, I can >> see all the classes of the DOM, still I can't figure out just how much >> of the memory they cost in total. > > Did you try the memory profiler from Eclipse's TPTP? How did it go? Well, maybe I didn't see something, but it basically leaves me with saying "Most memory is used by class char[]". Well thanks! I can then manually check what references to those char[]s but I don't think I'm going to do that a 150 000 times. Moritz >> The point I am trying to make is that this concept does not exist with >> regard to /referenced/ objects. The only size you can reasonably well >> detect is the size of the object itself. Hm, people keep making this point. > Point taken. That was never the question, but more: > General XML editor. I have to look into it some more, but 20/1 Memory > Usage/Document Size seems a little much. Maybe that's what they mean by > "Java is memory intensive" No, it isn't. "They" are referring to the JVM footprint. The overhead you're talking about is DOM overhead, not Java. SignatureLew >> General XML editor. I have to look into it some more, but 20/1 Memory >> Usage/Document Size seems a little much. Maybe that's what they mean by [quoted text clipped - 3 lines] > > The overhead you're talking about is DOM overhead, not Java. Well, I meant the general style that comes with Java, for example using the collection classes (HashMaps) which will add their share to memory usage. I guess I'm still surprised that everything in the implementation seems ok/necessary and still 1 MB blows up to 20 when loading a file. Moritz >>> General XML editor. I have to look into it some more, but 20/1 >>> Memory Usage/Document Size seems a little much. Maybe that's what >>> they mean by "Java is memory intensive" Lew wrote: >> No, it isn't. "They" are referring to the JVM footprint. >> The overhead you're talking about is DOM overhead, not Java. > Well, I meant the general style that comes with Java, for example using > the collection classes (HashMaps) which will add their share to memory > usage. It would be different in what other language? > I guess I'm still surprised that everything in the implementation seems > ok/necessary and still 1 MB blows up to 20 when loading a file. That's the price of all those links between nodes, and all that metadata, and whatnot. It's not the language, it's the algorithm. This is still not what "they" mean by "Java needs a lot of memory." This is what "they" mean by "DOM needs a lot of memory." There's no question that the memory cost is high. If that's a problem, consider SAX or StAX parsing instead - it's much more PARSimonious. Cover what you need in one pass and it's also extremely fast. SignatureLew "Moritz Wissenbach" <m._N0_SPM_wissenbach@gmx.de> said: >I have an application (xml-editor) that uses up an unreasonable amount of >java heap space. >I have tried profiling with tptp (eclipse) and jmap/jhat, and they do give >me memory histograms and instance counts, but what I REALLY want is to see >the total size of a classes instances INCLUDING the referenced objects. That looks pretty much what Yourkit does (among other things). http://www.yourkit.com/ (disclaimer: I have no affiliation with the company) SignatureWolf a.k.a. Juha Laiho Espoo, Finland (GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++ "...cancel my subscription to the resurrection!" (Jim Morrison) > Is there a profiler that will traverse the heap and find the "recursive" > size of objects? YourKit Java Profiler http://www.yourkit.com/download/ has exactly this feature BR, Vladimir > That looks pretty much what Yourkit does (among other things). > > http://www.yourkit.com/ > YourKit Java Profiler http://www.yourkit.com/download/ has exactly > this feature Hey, thanks a lot guys, I'll check that out! Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2009 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.