Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / November 2007

Tip: Looking for answers? Try searching our database.

read, write Objects via Sockets

Thread view: 
tricky - 03 Nov 2007 11:22 GMT
Hi

I have big problem with memory leak (heap size usage). I send object via
Sockets using object input, output stream.
The problem occurs on the client side when i receive an object

public class My implements Serializable{

/** Creates a new instance of My */
public My() {
}

private String newField =
"ewqeqwewqewdsadsadasdsadsadsadsadadsadsadasdsadsadsadasdasdasdasdasdadsadsasddsadasdsadsa";

public String s(){
return this.newField;
}

}

Server side !!!

ServerSocket s = new ServerSocket(1111);
Socket sv = s.accept();

ObjectOutputStream oos = new ObjectOutputStream(sv.getOutputStream());

while(true){

My m = new My();

oos.writeObject(m);
oos.flush();

m=null;

}
clietn side !!

while(true){

My x = (My)oos.readObject();

System.out.println("pdu size:"+x.s());
x = null;
}

The problem is that memory goes up each time i receive My object. Heap
memory goes up and up and gc() cannot clean memory.

But what is the most funny part . When i'm using insted of My class just
simple byte [] array and send via Socket also using object, input output
stream the problem doesn't occure !. But byte array is also serialized by
default .
Profiler in netbeans shows that it gowas up and down - so the way i would
like it to go :D
What may be the cause of this problem ? implementing Serializable or write,
read object methods have some problems ?
Gordon Beaton - 03 Nov 2007 12:03 GMT
> I have big problem with memory leak (heap size usage). I send object
> via Sockets using object input, output stream.
[...]
> The problem is that memory goes up each time i receive My object. Heap
> memory goes up and up and gc() cannot clean memory.
[quoted text clipped - 3 lines]
> input output stream the problem doesn't occure !. But byte array is
> also serialized by default .

The object stream caches every object you send through it. If you sent
many different objects, the cache grows. That's what you see when you
send your "My" objects.

When you send the same object a second or subsequent times, only a
handle is sent so the receiver can recreate the "same" object. I'll
guess you sent the *same* byte[] each time when you tested that, or
you would have seen the same growth.

You can clear the cache by resetting the stream, but one object sent
both before and after reset will appear at the receiver as two
distinct objects.

/gordon

--
tricky - 03 Nov 2007 15:52 GMT
I think there is a bug in writeObject or readObject!

So i decided to use ByteArrayOutputStream to change object into byte array
and send via object outputstream using write(byte,offset,len) and it works
fine !
So probably this caching stuff or something else is buggy !
Lew - 03 Nov 2007 16:10 GMT
> I think there is a bug in writeObject or readObject!

Strong words, considering that you've presented no evidence that you even have
a memory leak, and that the observed behavior is consistent with the design
parameters for the classes in question.

> So i [sic] decided to use ByteArrayOutputStream to change object into byte array
> and send via object outputstream using write(byte,offset,len) and it works
> fine !
> So probably this caching stuff or something else is buggy !

Almost certainly not.

A competent programmer blames the following, in order:

1.  Their own code.
2.  Their own code.
3.  Their own design.
4.  Their own code.
5.  Interactions with third-party libraries due to their own misunderstanding.
6.  Their own code.
7.  The platform, but only after very, very, very extensive testing and
incontrovertible evidence.

Not only have you not presented incontrovertible evidence of a bug, you have
presented no evidence at all of a bug, nor even of a problem.

Signature

Lew

Arne Vajhøj - 03 Nov 2007 18:47 GMT
>> I think there is a bug in writeObject or readObject!
>
> Strong words, considering that you've presented no evidence that you
> even have a memory leak, and that the observed behavior is consistent
> with the design parameters for the classes in question.

> Not only have you not presented incontrovertible evidence of a bug, you
> have presented no evidence at all of a bug, nor even of a problem.

It is well known that the object stream classes has a feature where
caching causes out of memory.

It is a feature. But it is not very well documented. And probably
unwanted by most users.

Arne
Roedy Green - 03 Nov 2007 19:25 GMT
>It is a feature. But it is not very well documented. And probably
>unwanted by most users.

The alternative would be to resend every object when there is yet
another reference to it, including string constants, which would
drastically bulk up the stream.

It might have been implemented with a max cache size on the sender
side.  It would then delete infrequently used items from its cache if
it were getting near the limit.

Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Arne Vajhøj - 03 Nov 2007 19:32 GMT
>> It is a feature. But it is not very well documented. And probably
>> unwanted by most users.
>
> The alternative would be to resend every object when there is yet
> another reference to it, including string constants, which would
> drastically bulk up the stream.

I have never seen an example in real life where this cache feature
was considered useful.

It is not even just the memory issue. It can also give some funny
effects with mutable objects.

Arne
Roedy Green - 03 Nov 2007 20:37 GMT
>It is not even just the memory issue. It can also give some funny
>effects with mutable objects.

The way it works now, if you change an object, and resend it, it does
not get sent.  A reference to the old copy gets sent.

If you used a some-times forgetting cache, that behaviour would be
indeterminate.  Sometimes you would get the old object, sometimes the
new. The rule would be, don't modify objects you are in the process of
serialising unless the changes don't matter.
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Gordon Beaton - 03 Nov 2007 20:43 GMT
> The way it works now, if you change an object, and resend it, it
> does not get sent. A reference to the old copy gets sent.

But that isn't an argument against caching per se, which could have
been implemented to detect when a sent object doesn't agree with the
cached one.

The caching itself provides a useful feature, making it possible to
preserve the topology of object references across the serialization.
If you send two distinct objects each with a reference to a common
third object, the recipient shouldn't receive four distinct objects.
Similarly, it makes it possible to correctly transfer structures
containing cyclic references.

/gordon

--
Roedy Green - 03 Nov 2007 21:16 GMT
>But that isn't an argument against caching per se, which could have
>been implemented to detect when a sent object doesn't agree with the
>cached one.

I guess you could just use equals for that determination.

The trouble is equals would often not detect a changed field.

But I suppose programmers could live with that, implementing equals if
it really mattered.  Perhaps a new interface would be needed so you
would not disturb equals semantics.

In most cases you really want the current value of all objects
transmitted.  The way it is now is an unfortunate accident of the
implementation.

It amazing that RPC works at all sitting atop a mechanism that does
not track changes.

Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Lothar Kimmeringer - 03 Nov 2007 20:52 GMT
> I have never seen an example in real life where this cache feature
> was considered useful.

------ snip
public class MyList implements Serializable
   public MyList next;
}

MyList a = new MyList();
MyList b = new MyList();

a.next = b;
b.next = a;
------ snap

Try to serialize that without a cache and you will run into problem.

Regards, Lothar
Signature

Lothar Kimmeringer                E-Mail: spamfang@kimmeringer.de
              PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
                questions!

Arne Vajhøj - 03 Nov 2007 21:00 GMT
>> I have never seen an example in real life where this cache feature
>> was considered useful.
[quoted text clipped - 12 lines]
>
> Try to serialize that without a cache and you will run into problem.

They keyword was "real life".

In real life I would expect a java.util.LinkedList to be send in
a single writeObject and I will assume that it would work fine.

Arne
Lasse Reichstein Nielsen - 04 Nov 2007 01:36 GMT
> In real life I would expect a java.util.LinkedList to be send in
> a single writeObject and I will assume that it would work fine.

The point was, I guess, to give an example of a cyclic data structure.

The "caching" feature of ObjectInputStream is really not about
caching, it's just nessecitated by ObjectOutputStream allowing
serialization of cyclic structures. Each object is only sent once,
and only received once.

If you send a data structure, or perhaps several of them, in one go,
the deserialization works as expected, and uses no more memory than
would be expected.

If you use the *same* Object{Output,Input}Stream to end objects as
part of different communication sessions, i.e., where something might
have happened to the objects in between, at either end, you *should*
reset the streams between sessions. Any "caching" that might occour
is most likely not what you want anyway. It's *not* a performance
feature.

/L
Signature

Lasse Reichstein Nielsen  -  lrn@hotpop.com
DHTML Death Colors: <URL:http://www.infimum.dk/HTML/rasterTriangleDOM.html>
 'Faith without judgement merely degrades the spirit divine.'

Lothar Kimmeringer - 04 Nov 2007 15:56 GMT
[Class with cyclic data-structure]

>> Try to serialize that without a cache and you will run into problem.
>
> They keyword was "real life".

Is
http://commons.apache.org/collections/api-release/org/apache/commons/collections
/buffer/CircularFifoBuffer.html

"real lify" enough?

Regards, Lothar
Signature

Lothar Kimmeringer                E-Mail: spamfang@kimmeringer.de
              PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
                questions!

Lew - 03 Nov 2007 20:15 GMT
Lew wrote:
>> Not only have you not presented incontrovertible evidence of a bug,
>> you have presented no evidence at all of a bug, nor even of a problem.

> It is well known that the object stream classes has a feature where
> caching causes out of memory.
>
> It is a feature. But it is not very well documented. And probably
> unwanted by most users.

It is common to see memory rise in a Java program without there being a memory
leak.  All the OP told us was that they were receiving an increase in memory
usage.  I did not see evidence that it was actually a leak.  Had they reported
an OOME that'd have been different.

I certainly learned something new about the object streams from this discussion.

Signature

Lew

tricky - 04 Nov 2007 09:45 GMT
I can send you a profiler view if you want :D.
The most importent thing for me is that i can avoid it by using different
methods
tricky - 04 Nov 2007 09:45 GMT
If you would search a little bit you would find thaht it's reported as a bug
! on february 2007 ! but it's not resovled yet
Esmond Pitt - 07 Nov 2007 02:55 GMT
> If you would search a little bit you would find thaht it's reported as a bug
> ! on february 2007 ! but it's not resovled yet

If you had searched a little more carefully you would have seen that bug
concerns a memory leak *at the sender* and *when you use
writeUnshared(). What is being discussed here is a possible memory leak
*at the receiver* when using readObject(). Nothing to do with it.
tricky - 07 Nov 2007 05:56 GMT
exaclty :P using readUnshered and writeUnshered and problem dissapered

>> If you would search a little bit you would find thaht it's reported as a
>> bug ! on february 2007 ! but it's not resovled yet
[quoted text clipped - 3 lines]
> What is being discussed here is a possible memory leak *at the receiver*
> when using readObject(). Nothing to do with it.
Lothar Kimmeringer - 03 Nov 2007 17:12 GMT
> So probably this caching stuff or something else is buggy !

So you are saying that a cache doing its job is regarded to be
buggy? I think I don't want to see your cache-implementations ;-)

Regards, Lothar
Signature

Lothar Kimmeringer                E-Mail: spamfang@kimmeringer.de
              PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
                questions!

Arne Vajhøj - 03 Nov 2007 15:54 GMT
> I have big problem with memory leak (heap size usage). I send object via
> Sockets using object input, output stream.
> The problem occurs on the client side when i receive an object

> oos.writeObject(m);

> My x = (My)oos.readObject();

> The problem is that memory goes up each time i receive My object. Heap
> memory goes up and up and gc() cannot clean memory.

That is a known "feature" of object streams.

You can code like this:

some loop:

            oos.writeObject(x);
            n++;
            if((n % 100) == 0) {
               oos.reset();
            }

Arne
Roedy Green - 03 Nov 2007 17:30 GMT
>The problem is that memory goes up each time i receive My object. Heap
>memory goes up and up and gc() cannot clean memory.

See http://mindprod.com/jgloss/serialization.html 
search for the phase "out of RAM".

The reason is the receiver has to maintain a catalog to all the
objects that have been received so far in the stream in case you send
a object later with a reference to one of them.

You can tell it to have an attack of amnesia with reset.

Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Knute Johnson - 03 Nov 2007 18:59 GMT
>> The problem is that memory goes up each time i receive My object. Heap
>> memory goes up and up and gc() cannot clean memory.
[quoted text clipped - 7 lines]
>
> You can tell it to have an attack of amnesia with reset.

Roedy:

I looked at your site and now I'm confused.  If you send a series of
Objects down the stream and call reset after sending each one, what
exactly happens on the receiving end?  Is there some reference kept by
the ObjectInputStream that will prevent the received objects from being
GCed?

Thanks,

Signature

Knute Johnson
email s/nospam/knute/

Roedy Green - 03 Nov 2007 19:28 GMT
On Sat, 03 Nov 2007 10:59:25 -0700, Knute Johnson
<nospam@rabbitbrush.frazmtn.com> wrote, quoted or indirectly quoted
someone who said :

>I looked at your site and now I'm confused.  If you send a series of
>Objects down the stream and call reset after sending each one, what
>exactly happens on the receiving end?  Is there some reference kept by
>the ObjectInputStream that will prevent the received objects from being
>GCed?

Hitting reset effectively closes and reopens the stream. Both sides
start from scratch.  Previously received objects would usually be
pointed to by the receiver, e.g. added to a Collection.  Any it does
not point to will be GCed. The sender won't expect the receiver to
have kept any objects from the previous batch.  So it will resend any
that are referenced again.
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Knute Johnson - 03 Nov 2007 20:27 GMT
> On Sat, 03 Nov 2007 10:59:25 -0700, Knute Johnson
> <nospam@rabbitbrush.frazmtn.com> wrote, quoted or indirectly quoted
[quoted text clipped - 12 lines]
> have kept any objects from the previous batch.  So it will resend any
> that are referenced again.

So if you send only new objects and reset between them, while slow,
there should be no risk of OOMException right?

Signature

Knute Johnson
email s/nospam/knute/

Roedy Green - 03 Nov 2007 20:40 GMT
On Sat, 03 Nov 2007 12:27:49 -0700, Knute Johnson
<nospam@rabbitbrush.frazmtn.com> wrote, quoted or indirectly quoted
someone who said :

>So if you send only new objects and reset between them, while slow,
>there should be no risk of OOMException right?

Yes.  I don't think it would be much slower than not using reset. I
think all reset does in tell both ends to flush their invisible cache
of previously transmitted objects.
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Roedy Green - 03 Nov 2007 21:08 GMT
On Sat, 03 Nov 2007 19:40:05 GMT, Roedy Green
<see_website@mindprod.com.invalid> wrote, quoted or indirectly quoted
someone who said :

>>So if you send only new objects and reset between them, while slow,
>>there should be no risk of OOMException right?
>
>Yes.  I don't think it would be much slower than not using reset. I
>think all reset does in tell both ends to flush their invisible cache
>of previously transmitted objects.

The other concern, is every time you call reset, all the CLASS
definitions are forgotten too and have to be resent.

You might get a feel for how this works by examining a
ObjectOutputStream with a hex viewer, salted with some unique strings.

Happily GZIP does a great job of compressing the stream.
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com



Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2009 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.