Java Forum / General / November 2007
read, write Objects via Sockets
tricky - 03 Nov 2007 11:22 GMT Hi
I have big problem with memory leak (heap size usage). I send object via Sockets using object input, output stream. The problem occurs on the client side when i receive an object
public class My implements Serializable{
/** Creates a new instance of My */ public My() { }
private String newField = "ewqeqwewqewdsadsadasdsadsadsadsadadsadsadasdsadsadsadasdasdasdasdasdadsadsasddsadasdsadsa";
public String s(){ return this.newField; }
}
Server side !!!
ServerSocket s = new ServerSocket(1111); Socket sv = s.accept();
ObjectOutputStream oos = new ObjectOutputStream(sv.getOutputStream());
while(true){
My m = new My();
oos.writeObject(m); oos.flush();
m=null;
} clietn side !!
while(true){
My x = (My)oos.readObject();
System.out.println("pdu size:"+x.s()); x = null; }
The problem is that memory goes up each time i receive My object. Heap memory goes up and up and gc() cannot clean memory.
But what is the most funny part . When i'm using insted of My class just simple byte [] array and send via Socket also using object, input output stream the problem doesn't occure !. But byte array is also serialized by default . Profiler in netbeans shows that it gowas up and down - so the way i would like it to go :D What may be the cause of this problem ? implementing Serializable or write, read object methods have some problems ?
Gordon Beaton - 03 Nov 2007 12:03 GMT > I have big problem with memory leak (heap size usage). I send object > via Sockets using object input, output stream. [...]
> The problem is that memory goes up each time i receive My object. Heap > memory goes up and up and gc() cannot clean memory. [quoted text clipped - 3 lines] > input output stream the problem doesn't occure !. But byte array is > also serialized by default . The object stream caches every object you send through it. If you sent many different objects, the cache grows. That's what you see when you send your "My" objects.
When you send the same object a second or subsequent times, only a handle is sent so the receiver can recreate the "same" object. I'll guess you sent the *same* byte[] each time when you tested that, or you would have seen the same growth.
You can clear the cache by resetting the stream, but one object sent both before and after reset will appear at the receiver as two distinct objects.
/gordon
--
tricky - 03 Nov 2007 15:52 GMT I think there is a bug in writeObject or readObject!
So i decided to use ByteArrayOutputStream to change object into byte array and send via object outputstream using write(byte,offset,len) and it works fine ! So probably this caching stuff or something else is buggy !
Lew - 03 Nov 2007 16:10 GMT > I think there is a bug in writeObject or readObject! Strong words, considering that you've presented no evidence that you even have a memory leak, and that the observed behavior is consistent with the design parameters for the classes in question.
> So i [sic] decided to use ByteArrayOutputStream to change object into byte array > and send via object outputstream using write(byte,offset,len) and it works > fine ! > So probably this caching stuff or something else is buggy ! Almost certainly not.
A competent programmer blames the following, in order:
1. Their own code. 2. Their own code. 3. Their own design. 4. Their own code. 5. Interactions with third-party libraries due to their own misunderstanding. 6. Their own code. 7. The platform, but only after very, very, very extensive testing and incontrovertible evidence.
Not only have you not presented incontrovertible evidence of a bug, you have presented no evidence at all of a bug, nor even of a problem.
 Signature Lew
Arne Vajhøj - 03 Nov 2007 18:47 GMT >> I think there is a bug in writeObject or readObject! > > Strong words, considering that you've presented no evidence that you > even have a memory leak, and that the observed behavior is consistent > with the design parameters for the classes in question.
> Not only have you not presented incontrovertible evidence of a bug, you > have presented no evidence at all of a bug, nor even of a problem. It is well known that the object stream classes has a feature where caching causes out of memory.
It is a feature. But it is not very well documented. And probably unwanted by most users.
Arne
Roedy Green - 03 Nov 2007 19:25 GMT >It is a feature. But it is not very well documented. And probably >unwanted by most users. The alternative would be to resend every object when there is yet another reference to it, including string constants, which would drastically bulk up the stream.
It might have been implemented with a max cache size on the sender side. It would then delete infrequently used items from its cache if it were getting near the limit.
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Arne Vajhøj - 03 Nov 2007 19:32 GMT >> It is a feature. But it is not very well documented. And probably >> unwanted by most users. > > The alternative would be to resend every object when there is yet > another reference to it, including string constants, which would > drastically bulk up the stream. I have never seen an example in real life where this cache feature was considered useful.
It is not even just the memory issue. It can also give some funny effects with mutable objects.
Arne
Roedy Green - 03 Nov 2007 20:37 GMT >It is not even just the memory issue. It can also give some funny >effects with mutable objects. The way it works now, if you change an object, and resend it, it does not get sent. A reference to the old copy gets sent.
If you used a some-times forgetting cache, that behaviour would be indeterminate. Sometimes you would get the old object, sometimes the new. The rule would be, don't modify objects you are in the process of serialising unless the changes don't matter.
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Gordon Beaton - 03 Nov 2007 20:43 GMT > The way it works now, if you change an object, and resend it, it > does not get sent. A reference to the old copy gets sent. But that isn't an argument against caching per se, which could have been implemented to detect when a sent object doesn't agree with the cached one.
The caching itself provides a useful feature, making it possible to preserve the topology of object references across the serialization. If you send two distinct objects each with a reference to a common third object, the recipient shouldn't receive four distinct objects. Similarly, it makes it possible to correctly transfer structures containing cyclic references.
/gordon
--
Roedy Green - 03 Nov 2007 21:16 GMT >But that isn't an argument against caching per se, which could have >been implemented to detect when a sent object doesn't agree with the >cached one. I guess you could just use equals for that determination.
The trouble is equals would often not detect a changed field.
But I suppose programmers could live with that, implementing equals if it really mattered. Perhaps a new interface would be needed so you would not disturb equals semantics.
In most cases you really want the current value of all objects transmitted. The way it is now is an unfortunate accident of the implementation.
It amazing that RPC works at all sitting atop a mechanism that does not track changes.
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Lothar Kimmeringer - 03 Nov 2007 20:52 GMT > I have never seen an example in real life where this cache feature > was considered useful. ------ snip public class MyList implements Serializable public MyList next; }
MyList a = new MyList(); MyList b = new MyList();
a.next = b; b.next = a; ------ snap
Try to serialize that without a cache and you will run into problem.
Regards, Lothar
 Signature Lothar Kimmeringer E-Mail: spamfang@kimmeringer.de PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)
Always remember: The answer is forty-two, there can only be wrong questions!
Arne Vajhøj - 03 Nov 2007 21:00 GMT >> I have never seen an example in real life where this cache feature >> was considered useful. [quoted text clipped - 12 lines] > > Try to serialize that without a cache and you will run into problem. They keyword was "real life".
In real life I would expect a java.util.LinkedList to be send in a single writeObject and I will assume that it would work fine.
Arne
Lasse Reichstein Nielsen - 04 Nov 2007 01:36 GMT > In real life I would expect a java.util.LinkedList to be send in > a single writeObject and I will assume that it would work fine. The point was, I guess, to give an example of a cyclic data structure.
The "caching" feature of ObjectInputStream is really not about caching, it's just nessecitated by ObjectOutputStream allowing serialization of cyclic structures. Each object is only sent once, and only received once.
If you send a data structure, or perhaps several of them, in one go, the deserialization works as expected, and uses no more memory than would be expected.
If you use the *same* Object{Output,Input}Stream to end objects as part of different communication sessions, i.e., where something might have happened to the objects in between, at either end, you *should* reset the streams between sessions. Any "caching" that might occour is most likely not what you want anyway. It's *not* a performance feature.
/L
 Signature Lasse Reichstein Nielsen - lrn@hotpop.com DHTML Death Colors: <URL:http://www.infimum.dk/HTML/rasterTriangleDOM.html> 'Faith without judgement merely degrades the spirit divine.'
Lothar Kimmeringer - 04 Nov 2007 15:56 GMT [Class with cyclic data-structure]
>> Try to serialize that without a cache and you will run into problem. > > They keyword was "real life". Is http://commons.apache.org/collections/api-release/org/apache/commons/collections /buffer/CircularFifoBuffer.html "real lify" enough?
Regards, Lothar
 Signature Lothar Kimmeringer E-Mail: spamfang@kimmeringer.de PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)
Always remember: The answer is forty-two, there can only be wrong questions!
Lew - 03 Nov 2007 20:15 GMT Lew wrote:
>> Not only have you not presented incontrovertible evidence of a bug, >> you have presented no evidence at all of a bug, nor even of a problem.
> It is well known that the object stream classes has a feature where > caching causes out of memory. > > It is a feature. But it is not very well documented. And probably > unwanted by most users. It is common to see memory rise in a Java program without there being a memory leak. All the OP told us was that they were receiving an increase in memory usage. I did not see evidence that it was actually a leak. Had they reported an OOME that'd have been different.
I certainly learned something new about the object streams from this discussion.
 Signature Lew
tricky - 04 Nov 2007 09:45 GMT I can send you a profiler view if you want :D. The most importent thing for me is that i can avoid it by using different methods
tricky - 04 Nov 2007 09:45 GMT If you would search a little bit you would find thaht it's reported as a bug ! on february 2007 ! but it's not resovled yet
Esmond Pitt - 07 Nov 2007 02:55 GMT > If you would search a little bit you would find thaht it's reported as a bug > ! on february 2007 ! but it's not resovled yet If you had searched a little more carefully you would have seen that bug concerns a memory leak *at the sender* and *when you use writeUnshared(). What is being discussed here is a possible memory leak *at the receiver* when using readObject(). Nothing to do with it.
tricky - 07 Nov 2007 05:56 GMT exaclty :P using readUnshered and writeUnshered and problem dissapered
>> If you would search a little bit you would find thaht it's reported as a >> bug ! on february 2007 ! but it's not resovled yet [quoted text clipped - 3 lines] > What is being discussed here is a possible memory leak *at the receiver* > when using readObject(). Nothing to do with it. Lothar Kimmeringer - 03 Nov 2007 17:12 GMT > So probably this caching stuff or something else is buggy ! So you are saying that a cache doing its job is regarded to be buggy? I think I don't want to see your cache-implementations ;-)
Regards, Lothar
 Signature Lothar Kimmeringer E-Mail: spamfang@kimmeringer.de PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)
Always remember: The answer is forty-two, there can only be wrong questions!
Arne Vajhøj - 03 Nov 2007 15:54 GMT > I have big problem with memory leak (heap size usage). I send object via > Sockets using object input, output stream. > The problem occurs on the client side when i receive an object
> oos.writeObject(m);
> My x = (My)oos.readObject();
> The problem is that memory goes up each time i receive My object. Heap > memory goes up and up and gc() cannot clean memory. That is a known "feature" of object streams.
You can code like this:
some loop:
oos.writeObject(x); n++; if((n % 100) == 0) { oos.reset(); }
Arne
Roedy Green - 03 Nov 2007 17:30 GMT >The problem is that memory goes up each time i receive My object. Heap >memory goes up and up and gc() cannot clean memory. See http://mindprod.com/jgloss/serialization.html search for the phase "out of RAM".
The reason is the receiver has to maintain a catalog to all the objects that have been received so far in the stream in case you send a object later with a reference to one of them.
You can tell it to have an attack of amnesia with reset.
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Knute Johnson - 03 Nov 2007 18:59 GMT >> The problem is that memory goes up each time i receive My object. Heap >> memory goes up and up and gc() cannot clean memory. [quoted text clipped - 7 lines] > > You can tell it to have an attack of amnesia with reset. Roedy:
I looked at your site and now I'm confused. If you send a series of Objects down the stream and call reset after sending each one, what exactly happens on the receiving end? Is there some reference kept by the ObjectInputStream that will prevent the received objects from being GCed?
Thanks,
 Signature Knute Johnson email s/nospam/knute/
Roedy Green - 03 Nov 2007 19:28 GMT On Sat, 03 Nov 2007 10:59:25 -0700, Knute Johnson <nospam@rabbitbrush.frazmtn.com> wrote, quoted or indirectly quoted someone who said :
>I looked at your site and now I'm confused. If you send a series of >Objects down the stream and call reset after sending each one, what >exactly happens on the receiving end? Is there some reference kept by >the ObjectInputStream that will prevent the received objects from being >GCed? Hitting reset effectively closes and reopens the stream. Both sides start from scratch. Previously received objects would usually be pointed to by the receiver, e.g. added to a Collection. Any it does not point to will be GCed. The sender won't expect the receiver to have kept any objects from the previous batch. So it will resend any that are referenced again.
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Knute Johnson - 03 Nov 2007 20:27 GMT > On Sat, 03 Nov 2007 10:59:25 -0700, Knute Johnson > <nospam@rabbitbrush.frazmtn.com> wrote, quoted or indirectly quoted [quoted text clipped - 12 lines] > have kept any objects from the previous batch. So it will resend any > that are referenced again. So if you send only new objects and reset between them, while slow, there should be no risk of OOMException right?
 Signature Knute Johnson email s/nospam/knute/
Roedy Green - 03 Nov 2007 20:40 GMT On Sat, 03 Nov 2007 12:27:49 -0700, Knute Johnson <nospam@rabbitbrush.frazmtn.com> wrote, quoted or indirectly quoted someone who said :
>So if you send only new objects and reset between them, while slow, >there should be no risk of OOMException right? Yes. I don't think it would be much slower than not using reset. I think all reset does in tell both ends to flush their invisible cache of previously transmitted objects.
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Roedy Green - 03 Nov 2007 21:08 GMT On Sat, 03 Nov 2007 19:40:05 GMT, Roedy Green <see_website@mindprod.com.invalid> wrote, quoted or indirectly quoted someone who said :
>>So if you send only new objects and reset between them, while slow, >>there should be no risk of OOMException right? > >Yes. I don't think it would be much slower than not using reset. I >think all reset does in tell both ends to flush their invisible cache >of previously transmitted objects. The other concern, is every time you call reset, all the CLASS definitions are forgotten too and have to be resent.
You might get a feel for how this works by examining a ObjectOutputStream with a hex viewer, salted with some unique strings.
Happily GZIP does a great job of compressing the stream.
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|