 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / General / August 2005"Virtual memory" framework for Java
Hi all, I am dealing with very large in-memory data structures. The data structures are accessed in different frequencies varying between every 30 seconds and few minutes. My first try was to zip the information to a file when un-needed, but it caused very high I/O wait on Solaris machine. What I really want is something like virtual memory - I want to be oblivious to where portions of the data structure are stored (in-memory or on disk) whenever I need to access the data, I want it to be loaded into memory. Do you know of such a framework for "virtual memory" in Java? Thanks > I am dealing with very large in-memory data structures. The data > structures are accessed in different frequencies varying between every [quoted text clipped - 5 lines] > be loaded into memory. > Do you know of such a framework for "virtual memory" in Java? Use java.nio to memory map a file. Be careful what you map though. If the file goes away or truncates you can get an asynchronous exception (InternalError), and on Linux the JVM actually crashes. Tom Hawtin  SignatureUnemployed English Java programmer http://jroller.com/page/tackline/ Hi, >> I am dealing with very large in-memory data structures. The data >> structures are accessed in different frequencies varying between every [quoted text clipped - 7 lines] > > Use java.nio to memory map a file. Wich class of the package do you mean exactly? I find nothing apropriate. My idea would be to use the java.lang.ref-Package to achieve some kind of virtual memory. Ciao, Ingo > > I am dealing with very large in-memory data structures. The data > > structures are accessed in different frequencies varying between every [quoted text clipped - 16 lines] > Unemployed English Java programmer > http://jroller.com/page/tackline/ Can you elaborate a little bit about the utilization of the memory mapped file for virtual memory as I described? >>> I am dealing with very large in-memory data structures. The >>> data structures are accessed in different frequencies varying [quoted text clipped - 19 lines] > Can you elaborate a little bit about the utilization of the memory > mapped file for virtual memory as I described? Does http://java.sun.com/j2se/1.4.2/docs/api/java/nio/MappedByteBuffer.html help? SignatureVirgil Hi zeus, > Can you elaborate a little bit about the utilization of the memory > mapped file for virtual memory as I described? Again: I'm not sure if the MappedByteBuffer is apropriate for what you want to do, since AFAICS it only maps bytes (and other native types) and not complete Objects! I think, something like java.lang.ref will be better since you are able to de-/serialize complete Objects using this. Ciao, Ingo ihomann_spam@web.de wrote in comp.lang.java.programmer: > Again: I'm not sure if the MappedByteBuffer is apropriate for what you > want to do, since AFAICS it only maps bytes (and other native types) and > not complete Objects! Objects can be serialized to bytes. SignatureAntti S. Brax Rullalautailu pitää lapset poissa ladulta http://www.iki.fi/asb/ http://www.cs.helsinki.fi/u/abrax/hlb/ [1385 messages expunged from folder "Spam"] Hi Antti, > ihomann_spam@web.de wrote in comp.lang.java.programmer: > [quoted text clipped - 3 lines] > > Objects can be serialized to bytes. OK, but isn't this overhead? When using a MappedByteBuffer, *everytime* you want to access an Object, it must be deserialized, and not only, when it is stored to disk! Ciao, Ingo ihomann_spam@web.de wrote in comp.lang.java.programmer: > Hi Antti, > [quoted text clipped - 9 lines] > you want to access an Object, it must be deserialized, and not only, > when it is stored to disk! Of course it is. But I was just pointing out an error in your statement. Anyway this discussion is useless since the original poster did not specify that he is serializing objects (his use of ZIP files indicates against it). SignatureAntti S. Brax Rullalautailu pitää lapset poissa ladulta http://www.iki.fi/asb/ http://www.cs.helsinki.fi/u/abrax/hlb/ [1385 messages expunged from folder "Spam"] Hi, >>>>Again: I'm not sure if the MappedByteBuffer is apropriate for what you >>>>want to do, since AFAICS it only maps bytes (and other native types) and [quoted text clipped - 8 lines] > Of course it is. But I was just pointing out an error in your > statement. I meant it is not adequate. I did not mean, that it is impossible. (-; Note, that we are both "Erbsenzähler" (*), as we say in germany. ;-) > Anyway this discussion is useless since the original > poster did not specify that he is serializing objects (his use > of ZIP files indicates against it). Well, I guess, zipping a serialized Object is more straightforward than zipping a non-serialized Object which must be serialized before zipping... :-) Anyway you are right, the discussion is useless since the OP does not tell us a bit more about his problem... Ciao, Ingo (*) That is a nice word for "Korinthenkacker" which (as http://dict.leo.org says) means "nitpicker", which - in my hears - sounds not so nice... or am I wrong? What do you say in Finland? ihomann_spam@web.de wrote in comp.lang.java.programmer: > (-; Note, that we are both "Erbsenzähler" (*), as we say in germany. ;-) > (*) That is a nice word for "Korinthenkacker" which (as > http://dict.leo.org says) means "nitpicker", which - in my hears - > sounds not so nice... or am I wrong? What do you say in Finland? It's unprintable. SignatureAntti S. Brax Rullalautailu pitää lapset poissa ladulta http://www.iki.fi/asb/ http://www.cs.helsinki.fi/u/abrax/hlb/ [1385 messages expunged from folder "Spam"] A lot to comment on. 1. When I say huge data structures I mean a lot of varying sized data structures from very small to few megabytes in the larger. When I'm saying a lot I refer to 1000s of such separate structures. 2. I don't see how weak (or other) references can help in this case. For as long as they have reference they will remain in memory and when they lose reference then anyway they will be removed (by gc). What I want is to load and unload from memory as needed. 3. When I say some are accessed frequently I mean every 30 seconds, and this applies the small data structures mainly. The larger structures are accessed every 10-15 minutes. 4. I tried the zipping on the larger data structures which are less frequently accessed and it caused the OS to stuck on IO wait when zipping and unzipping (gzip input/output stream) 5. Serialization and deserialization are very expensive, however is there any clean way to avoid them? Hi zeus, > A lot to comment on. > 1. When I say huge data structures I mean a lot of varying sized data > structures from very small to few megabytes in the larger. When I'm > saying a lot I refer to 1000s of such separate structures. OK. A "good" solution should be able to deal with this. > 2. I don't see how weak (or other) references can help in this case. > For as long as they have reference they will remain in memory and when > they lose reference then anyway they will be removed (by gc). What I > want is to load and unload from memory as needed. Well, AFAIK there *are* possibilities to get informed when a WeakRef will be 'removed'. IIRC, ReferenceQueue does this (but working with finalize (which works much better than it is said to) may work as well). So, when a the WeakRef is removed, you store the object to disk, and when someone tries to access it afterwards, you reload it. I think, this should work! > 3. When I say some are accessed frequently I mean every 30 seconds, and > this applies the small data structures mainly. The larger structures > are accessed every 10-15 minutes. It should be no greater problem to "count" how often/frequent an object is accessed and to primarily suspend objects to disk that are accessed in a low frequency. > 4. I tried the zipping on the larger data structures which are less > frequently accessed and it caused the OS to stuck on IO wait when > zipping and unzipping (gzip input/output stream) That problem should not occur when using this self-implemented solution. > 5. Serialization and deserialization are very expensive, however is > there any clean way to avoid them? You can implement your own methods for doing this, but I cannot say in how far this is faster that the java-build-in serialisation. Ciao, Ingo zohar@sheernetworks.com wrote in comp.lang.java.programmer: > I am dealing with very large in-memory data structures. What does very large mean? > The data > structures are accessed in different frequencies varying between every > 30 seconds and few minutes. My first try was to zip the information to > a file when un-needed, but it caused very high I/O wait on Solaris > machine. Did it cause _IO wait_ or wait on the _compression algorithm_ used in ZIP files? Do you have several ZIP files or one large? How did you organize the ZIP file? Did you try different directory hierarchies? Have you tried plain files instead of ZIP files? You are not giving us enough information to help you efficiently. SignatureAntti S. Brax Rullalautailu pitää lapset poissa ladulta http://www.iki.fi/asb/ http://www.cs.helsinki.fi/u/abrax/hlb/ [1385 messages expunged from folder "Spam"] Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.