 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / General / May 2007Memory question
I have a quick question....I am having an app built and am doing some test on it to see how it runs, etc. Java looks to eat up LOTS of memory, but here is what I see. I open the app and then create a profile...another window pops up to enter the info...of course the memory goes up some. Well when I close that window out...the memory never goes down. As a matter of fact if I create yet another profile...memory usage goes up even higher...is this normal... JediKnight2 wrote On 05/02/07 16:43,: > I have a quick question....I am having an app built and am doing some > test on it to see how it runs, etc. Java looks to eat up LOTS of [quoted text clipped - 3 lines] > never goes down. As a matter of fact if I create yet another > profile...memory usage goes up even higher...is this normal... Not enough detail to tell "normal" from "abnormal," but I'll offer a few thoughts: First, how are you measuring the memory usage, and from what vantage point? Few applications -- including JVM's -- actually release memory once they've obtained it from the operating system; they're more likely to hang onto it for possible re-use inside the app. (The fact that the app needed N megabytes at some point in its history is taken as evidence that it's likely to need N again.) So if you're looking at memory usage from the O/S' perspective, you'll seldom see processes shrink. Second, have you arranged to dispose() the windows when you're done with them, or are you just dropping them on the floor? If you just forget about them after they close they do not become collectible garbage: Swing has them sitting there, undisplayed but still instantiated and ready to be redisplayed at a moment's notice. If you are accustomed to using DISPOSE_ON_CLOSE or EXIT_ON_CLOSE this fact may have escaped you, but if you've selected one of the other window-closing actions and have forgotten to call dispose() explicitly, you may be piling up heaps and heaps of unused and semi-forgotten JFrames.  SignatureEric.Sosman@sun.com > JediKnight2 wrote On 05/02/07 16:43,: > [quoted text clipped - 33 lines] > -- > Eric.Sos...@sun.com I am just looking at the memory usage in task manager. Basically the program connects to a DB on the backend to input data from the form. Taskbar is showing up to 336,000 K of memory used...I will have to look into how it is dropping the windows...What I am thinking is maybe this thing should have been done in C++ or something that wouldn't use as much memory. One thing that drew me to Java was the ability to use it across all OS platforms... > > JediKnight2 wrote On 05/02/07 16:43,: > [quoted text clipped - 41 lines] > as much memory. One thing that drew me to Java was the ability to use > it across all OS platforms... And now that I look at the program and what is happening...it has to be that the data isnt being disposed of...I can simply open a create new profile window...close it then open another one...close it open another one close it and the memory just keeps climbing... JediKnight2 wrote On 05/02/07 20:00,: >>JediKnight2 wrote On 05/02/07 16:43,: >> [quoted text clipped - 41 lines] > as much memory. One thing that drew me to Java was the ability to use > it across all OS platforms... Since you're measuring the memory usage from the point of view of the O/S, the stuff in my first paragraph applies: once the JVM (or pretty much any other application) gets some memory from the O/S, it usually hangs onto it until death do them part. From the O/S' perspective all the memory is "used" even if (at some particular moment) the bulk of it is "free space" as far as the JVM is concerned. As for figuring out how the JVM is using the memory, there are Java memory profilers that can produce a lot of information about such things. SignatureEric.Sosman@sun.com > Since you're measuring the memory usage from the point > of view of the O/S, the stuff in my first paragraph applies: [quoted text clipped - 3 lines] > even if (at some particular moment) the bulk of it is "free > space" as far as the JVM is concerned. Another data point. I'm working on a JavaMail app with a small (28,000 message) collection of mail in an mbox as volume test data. Some of the messages have large attachments (lots of images). This all runs under a Linux (Fedora Core 6) system with 256 MB and JDK 1.4.2_03. Unsurprisingly, it hit memory limits and stopping, so I kept pushing the limit up via the -Xmx, eventually getting to 350m. Very much to my surprise nothing barfed, though when its processing the larger mails things slow do down as it starts swapping like mad. I've watched what's going on via the top utility. The assigned memory eventually peaks out at 400 MB and then declines, ending the run at around 208 MB. Releasing the memory surprised me. I had GC monitoring on at one point and the memory release was connected with the most drastic garbage collection modes. I'd be interested in knowing if this only happens with Linux/POSIX type OSen and/or if it only occurs when RAM is oversubscribed and swapping is taking place. Signaturemartin@ | Martin Gregorie gregorie. | Essex, UK org | Martin Gregorie wrote On 05/03/07 13:45,: > [... processing mail with large attachments ...] > Very much to my surprise nothing barfed, though when its processing the [quoted text clipped - 4 lines] > at one point and the memory release was connected with the most drastic > garbage collection modes. By "assigned memory" do you mean the process' total virtual memory, or the subset resident in physical RAM? The former only shrinks by explicit action on the part of the program ("Dear O/S: Take back your mink!"), while the latter will fluctuate upwards and downwards as the system responds to changing demands for memory. It's certainly possible that the JVM explicitly releases virtual memory under some conditions, but few applications find it worthwhile to do so. If there's a substantial amount of memory managed via JNI (maybe plugins for the attachments?), that's perhaps a more likely memory-releaser. SignatureEric.Sosman@sun.com > Martin Gregorie wrote On 05/03/07 13:45,: >> [... processing mail with large attachments ...] [quoted text clipped - 15 lines] > It's certainly possible that the JVM explicitly > releases virtual memory under some conditions, but few See the -XX:MaxHeapFreeRatio=70 option for HotSpot JVMs http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp If, after a GC, the heap would have more than 70% free space, then the allocated heap size is reduced by returning memory to the OS. Mark Thornton >> Martin Gregorie wrote On 05/03/07 13:45,: >>> [... processing mail with large attachments ...] [quoted text clipped - 22 lines] > If, after a GC, the heap would have more than 70% free space, then the > allocated heap size is reduced by returning memory to the OS. Makes sense in this case. If the CG runs while a small message is being processed after one of more huge ones you could well see over 70% free space in the heap. Signaturemartin@ | Martin Gregorie gregorie. | Essex, UK org | > By "assigned memory" do you mean the process' total > virtual memory, or the subset resident in physical RAM? Its my understanding that top shows total virtual memory. That fits with what I was seeing: I only have 256 MB of physical RAM and top reported the app+jvm as using 400 MB at times. To add to the memory over subscription, my application validates the sending host name by looking it up on DNS (it uses dnsjava for that and checks the domain's MX list as well as the domain name). I run a local DNS as a caching name server, so that was also active while all this was going on. I'm not knowingly using JNI in this application: the only nonstandard packages are javamail (which uses jaf) and dnsjava. > The former only shrinks by explicit action on the part > of the program ("Dear O/S: Take back your mink!"), while > the latter will fluctuate upwards and downwards as the > system responds to changing demands for memory. Yes, that's what I was expecting to see. > It's certainly possible that the JVM explicitly > releases virtual memory under some conditions, but few > applications find it worthwhile to do so. As I said, I wonder if it would start doing that once its working page set starts being swapped out to disk. It could help to improve performance if the CG was to return garbage-collected pages. Reducing the size of virtual memory offers a chance of fitting the entire program inside physical RAM again. This should tend to reduce page swapping. Signaturemartin@ | Martin Gregorie gregorie. | Essex, UK org | Martin Gregorie wrote On 05/03/07 17:05,: >> It's certainly possible that the JVM explicitly >>releases virtual memory under some conditions, but few [quoted text clipped - 5 lines] > the size of virtual memory offers a chance of fitting the entire program > inside physical RAM again. This should tend to reduce page swapping. (See also Mark Thornton's response, describing how to get HotSpot to release memory to the O/S.) Releasing virtual memory doesn't by itself do a lot to reduce paging pressure. If the system as a whole is short on memory, the stuff that goes out to disk will (should) be the stuff the programs aren't actively using. If Process P owns 500MB but is only actively using 100MB, the other 400MB should migrate to the swap device and sit there idly; the only effect of releasing the VM would be to make more swap space available, not to decrease the paging rates. Of course, P might well change its behavior if it knew it were running with just 100MB instead of 500MB -- for example, it would touch the other 400MB "not at all" instead of "seldom." But even that effect could be achieved without actually giving up the 400MB: just stop touching it for a while, and it will migrate out to swap. It's a lot easier to stop touching the extra space if you actually let go of it (because the O/S will slap your wrist if you get careless), but it's the change in memory access patterns, not the fact of jettisoning the VM, that affects paging behavior. Also, it's difficult to make system-wide policy decisions from inside the confines of a single process. P might well observe that paging rates are rising, but what should P's response be? P is ignorant of what else is running on the box; paging rates might be going up because Process Q has become greedy. Should P flush its caches and go on a memory diet for Q's sake? That would seem to depend on the relative importance of P and Q in the esteem of the system operator, something neither P nor Q has much knowledge of. It seems to me that most decisions P might try to make in this regard would be uninformed guesswork. The option Mark Thornton mentions is, in its way, a means of telling P something about its relative importance in the grand scheme of things, and helping it with its decision of how to consume resources. (Although it doesn't seem that HotSpot actually monitors paging rates to make the decision; it instead acts when it perceives an embarasse de richesse.) SignatureEric.Sosman@sun.com > Martin Gregorie wrote On 05/03/07 17:05,: >>> It's certainly possible that the JVM explicitly [quoted text clipped - 47 lines] > HotSpot actually monitors paging rates to make the decision; > it instead acts when it perceives an embarasse de richesse.) If a lot of objects are marked for deletion (i.e. not about to be reused until more objects are created) the remaining memory could get extremely fragmented and sparse. If the GC is going 6to do anything worthwhile with a sparse heap, it needs to compact the active objects, which has the effect of reducing the number of active pages and hence would improve paging performance. At this point I'm speculating, so its probably wise to drop this sub thread. Signaturemartin@ | Martin Gregorie gregorie. | Essex, UK org | > If a lot of objects are marked for deletion (i.e. not about to be reused > until more objects are created) the remaining memory could get > extremely fragmented and sparse. Huh? The only time memory fragmentation is important is when unallocated memory gets fragmented, which doesn't happen in Java's way of things, at least not with most of the GC algorithms in Sun's VMs. Furthermore, these collectors don't mark dead objects for deletion, they copy live ones. And dead ones are never re-used, not when more objects or created or any other time. > If the GC is going 6to do anything worthwhile with a sparse heap, it needs to compact the active objects, > which has the effect of reducing the number of active pages and hence > would improve paging performance. Not true. The page from which objects are copied is active for a while, and the page to which the objects are copied is also active for a while. Even before the GC runs, pages that haven't been used but are not collectible can become inactive, they will swap out and not swap in again until needed. If they aren't needed for a while, they won't hurt paging performance. Generally speaking whether memory is compacted has no effect on paging. Locality of reference does, but that is not guaranteed by a compact heap. > At this point I'm speculating, so its probably wise to drop this sub > thread. Not until the misconceptions are cleared up. SignatureLew > Huh? The only time memory fragmentation is important is when > unallocated memory gets fragmented, which doesn't happen in Java's way > of things, at least not with most of the GC algorithms in Sun's VMs. Wrong term - I should have said sparse. If a lot of large objects have become unreferenced as they go out of scope the heap is sparse for active objects. > Furthermore, these collectors don't mark dead objects for deletion, they > copy live ones. Yes, I should have said unreferenced objects. > And dead ones are never re-used, not when more objects > or created or any other time. Of course, but the space they occupied in heap space will be reused as active objects are collected at one end of the heap and unreferenced memory accumulates at the other prior to being amalgamated into a free space pool. > Not true. The page from which objects are copied is active for a while, > and the page to which the objects are copied is also active for a while. True, but once the GC has finished its work paging should be reduced somewhat simply because on average each page turn will now tend to fetch more active objects. If active objects now occupy a much smaller part of the heap space its possible that each will be paged into real memory as a side-effect of GC operation and not paged out again for some time. > Even before the GC runs, pages that haven't been used but are not > collectible can become inactive, they will swap out and not swap in > again until needed. Thats clear, but performance could be hurt if this unallocated heap space is retained in the process' virtual memory space but is paged out into swap space. It will have to be read in again before a new object is allocated to it. Whereas, if the paged out virtual memory is released by the GC then new objects can be allocated to newly claimed pages without needing to touch disk unless/until they are paged out. > Locality of reference does, but that is not guaranteed by a compact heap. But, doesn't the CG relocate active objects when its compacting the heap? That's going to affect locality: if active objects are not moved in virtual memory the GC can only release complete pages as they become entirely vacant. Additionally its operation would be greatly affected by the page size, which can vary wildly between operating systems. Signaturemartin@ | Martin Gregorie gregorie. | Essex, UK org | > Wrong term - I should have said sparse. If a lot of large objects have > become unreferenced as they go out of scope the heap is sparse for > active objects. Which doesn't necessarily affect much. Active objects are referred to directly so their contiguity has little to do with memory access speed. Even if objects are next to each other in memory, there is no guarantee that the program will use them in near time periods. Even if they are far apart in memory is no guarantee that the program will need them at the same time and run into swap trouble. In fact, the most likely scenario is that active objects will live and die more or less at the same time, so the sparseness about which you're concerned is unlikely to be much of a factor at all. Of course, all of this is speculation without actual meory profiling, and different applications will behave differently. Still, there is little reason to expect "sparseness" to have much effect or even to be much present. Lew wrote: >> The page from which objects are copied is active for a >> while, and the page to which the objects are copied is also active for >> a while. > True, but once the GC has finished its work paging should be reduced > somewhat simply because on average each page turn will now tend to fetch > more active objects. If active objects now occupy a much smaller part of > the heap space its possible that each will be paged into real memory as > a side-effect of GC operation and not paged out again for some time. Yes, I see that. It's just that if you're in a situation where memory is swapping like mad, you probably have a whole lot of things using memory, and most of them likely outside the instance of the JVM, so the swap situation is likely to remain bad after GC cycles, too. >> Even before the GC runs, pages that haven't been used but are not >> collectible can become inactive, they will swap out and not swap in >> again until needed. > Thats clear, but performance could be hurt if this unallocated heap > space is retained in the process' virtual memory space but is paged out > into swap space. It will have to be read in again before a new object is > allocated to it. Whereas, if the paged out virtual memory is released by > the GC then new objects can be allocated to newly claimed pages without > needing to touch disk unless/until they are paged out. Trivially true, but probably not much help when your physical RAM is maxed. At that point you are subject to mostly active memory requirements, and much of it likely not in the Java program, so GC is unlikely to help much. Certainly there is little reason to count on it helping. >> Locality of reference does, but that is not guaranteed by a compact heap. > But, doesn't the CG relocate active objects when its compacting the > heap? That's going to affect locality: if active objects are not moved > in virtual memory the GC can only release complete pages as they become > entirely vacant. Additionally its operation would be greatly affected by > the page size, which can vary wildly between operating systems. Locality of reference is determined by the algorithm patterns, not by heap compactness. For example, the heap might contain an object at the beginning that is used with an object at the other end, far enough apart that they are in separate memory pages even though they are referentially associated. This is despite the heap being completely compact. There just is no general correlation between how compact the heap is and how referentially associated near objects are. This becomes more of an issue the more memory the program is actively using, thus the more pages needed to hold everything. AFAIK, the JVM makes no effort to determine how "close" objects are algorithmically when copying its generations. SignatureLew > Of course, all of this is speculation without actual meory profiling, > and different applications will behave differently. Still, there is > little reason to expect "sparseness" to have much effect or even to be > much present. Agreed, and I'm conscious that my take on this may be over-influenced by the old ICL VME2900 virtual memory arrangement (in brief: it had a 3 level structure (disk, main RAM, one code page at at time in the processor cache together with a set of associatively addressed variables and pointers). Code locality mattered hugely in that architecture and paging algorithms could be tuned on a per-program basis. > It's just that if you're in a situation where memory is swapping like > mad, you probably have a whole lot of things using memory, and most of > them likely outside the instance of the JVM, so the swap situation is > likely to remain bad after GC cycles, too. Yes, in the fully general case. In the case I initially quoted I only had three busy processes: - the 'top' utility whose memory doesn't change appreciably during a run and is fairly small (no GUI). - the JavaMail application which is claiming and releasing heap in large chunks on a per-message basis but is otherwise quite static - named which is handling around 1 DNS lookup per message and caching the results. Its memory churn appears minimal by comparison because its only caching a new lookup result (which is quite small) each time it gets a hit. Again, named isn't a large process. > Trivially true, but probably not much help when your physical RAM is > maxed. At that point you are subject to mostly active memory > requirements, and much of it likely not in the Java program, so GC is > unlikely to help much. My point here is that if the allocation process decided to reuse addresses that are mapped onto a swapped out page, that page will be swapped in by the OS, which has no clue what its about to be used for, but if the page associated with that address range has been released the swap in is avoided. This could be quite significant if the application, like mine, has large amounts of heap churn. > There just is no general correlation between how compact the > heap is and how referentially associated near objects are. Agreed. > This becomes more of an issue the more memory the program is actively > using, thus the more pages needed to hold everything. Agreed. > AFAIK, the JVM > makes no effort to determine how "close" objects are algorithmically > when copying its generations. I wasn't assuming that it would, though this would be side effect of objects being moved in virtual memory by the GC: persistent objects would tend be grouped together by a series of GC runs. Signaturemartin@ | Martin Gregorie gregorie. | Essex, UK org | [...] > Second, have you arranged to dispose() the windows > when you're done with them, or are you just dropping them [quoted text clipped - 7 lines] > call dispose() explicitly, you may be piling up heaps and > heaps of unused and semi-forgotten JFrames. JediKnight2, if you have MS-Visual-Studio installed on your Windows system, then you can use its "Spy++" tool (file name "spyxx.exe") to find undisposed JFrames. Spy++ will show you a tree-view of processes, threads and window-handles. [Don't forget to refresh (key F5) the Spy from time to time.] In the process list first find your Java process. Within in this process find the AWT-thread (the only one which has window-handles). Look how many window-handles you have there. You should find very few top-level-windows (one per Frame or Dialog). If there are much more, then you probably have the problem of undisposed JFrames as Eric Sosman described above. SignatureThomas Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.