Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / November 2007

Tip: Looking for answers? Try searching our database.

memory management

Thread view: 
josh - 31 Oct 2007 10:22 GMT
Hi, I have the following problem:

in a while loop I get from a db some records and in the same time
I put them with an insert preparedStatement in
a destination db but the jave process increase the memory until it
goes in out of memory.

How can I optimize that? also If I increase Tomcat java memory when I
process more than 500 records
tha java memory is already at 700mb of usage!

Please help
GArlington - 31 Oct 2007 10:46 GMT
> Hi, I have the following problem:
>
[quoted text clipped - 8 lines]
>
> Please help

You can loop through the records (i.e. fetch them one-by-one), reuse
the same variables and run gc() after each (or few consecutive) loop
iterations.
Lew - 31 Oct 2007 14:14 GMT
josh wrote:
>> in a while loop I get from a db some records and in the same time
>> I put them with an insert preparedStatement in
[quoted text clipped - 4 lines]
>> process more than 500 records
>> tha java memory is already at 700mb of usage!

> You can loop through the records (i.e. fetch them one-by-one), reuse
> the same variables and run gc() after each (or few consecutive) loop
> iterations.

It is bad advice to suggest using the gc() call.

It is also usually better to allocate new objects in the loop than to hang on
to one for multiple iterations.

To the OP:

Provide an SSCCE.
<http://www.physci.org/codes/sscce.html>

You have a memory leak.  It is in your code.  The trick is to find from where.

Without an example that recreates the error we likely cannot help in detail.
Candidates are objects that are not abandoned, contrary to GArlington's
suggestion.  You are probably not close()ing a ResultSet or a Connection,
although that usually would cause problems before OutOfMemory (OOM).

GArlington's suggestion to process records one by one is sound.  Are you
storing many records' worth of information at once?

You probably need to refactor some logic into separate methods and classes.

The SSCCE will show us.

Signature

Lew

Roedy Green - 31 Oct 2007 19:05 GMT
>It is also usually better to allocate new objects in the loop than to hang on
>to one for multiple iterations.

Why is that?  
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Lew - 31 Oct 2007 23:47 GMT
>> It is also usually better to allocate new objects in the loop than to hang on
>> to one for multiple iterations.
>
> Why is that?  

On account of the garbage collector is optimized for short-lived objects, and
making long-lived ones risks having inter-generational references.  In
addition, the optimizer is able to optimize away the object allocation
altogether in many cases.  The cost of short-time object memory allocation and
GC is much less than most people suppose.

I have been reading article after article explaining this and recommending
that holding on to objects is not always helpful, and can even be harmful.

Also, extending an object's life artificially widens the scope of the
variable, which is against good code engineering maxims.

Naturally there are situations where it's better to keep an object than
reallocate, but the use of widened object scope to "optimize" turns out to be
an antipattern.

Articles that shed some light on this and related topics:
<http://www.ibm.com/developerworks/java/library/j-jtp01274.html?S_TACT=105AGX02&S
_CMP=EDU
>

<http://www.ibm.com/developerworks/java/library/j-jtp09275.html>
<http://www.ibm.com/developerworks/java/library/j-jtp04223.html>

Signature

Lew

Wojtek - 05 Nov 2007 15:51 GMT
Lew wrote :
> On account of the garbage collector is optimized for short-lived objects, and
> making long-lived ones risks having inter-generational references.  In
> addition, the optimizer is able to optimize away the object allocation
> altogether in many cases.  The cost of short-time object memory allocation
> and GC is much less than most people suppose.

Sigh.

More code to refactor...

I wonder, is there a tool (Eclipse plugin?) which scans code and
reduces variable scope to the least possible visibility.

Signature

Wojtek :-)

Daniel Pitts - 05 Nov 2007 19:52 GMT
> Lew wrote :
>> On account of the garbage collector is optimized for short-lived
[quoted text clipped - 9 lines]
> I wonder, is there a tool (Eclipse plugin?) which scans code and reduces
> variable scope to the least possible visibility.

The IDE I use will do that (IntelliJ IDEA).  You can get a free trial on
their website.  You would want to use the Analysis tool.  There may be
other tools out there for this, but I don't know about them.

Signature

Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>

Roedy Green - 05 Nov 2007 18:46 GMT
>I have been reading article after article explaining this and recommending
>that holding on to objects is not always helpful, and can even be harmful.

Would this also apply to objects > 1 MB?
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Thomas Kellerer - 31 Oct 2007 10:56 GMT
josh, 31.10.2007 10:22:
> Hi, I have the following problem:
>
[quoted text clipped - 6 lines]
> process more than 500 records
> tha java memory is already at 700mb of usage!

My guess is that you are buffering the retrieved rows somehow (or
keeping references to them). You will need to show us your code.

Thomas
josh - 31 Oct 2007 11:14 GMT
> josh, 31.10.2007 10:22:
>
[quoted text clipped - 13 lines]
>
> Thomas

ok I post the code:

st_from = conn.createStatement();

rs_from = st_from.executeQuery(query_from);

ps_into = conn_t.prepareStatement(insert_into);

int i_out_record = 0;
int i_in_record = 0;

while(rs_from.next())
{

i_out_record++;

            out_record += "SELECT su record (" + i_out_record + ")";
            in_record += "INSERT su record (";

            row_error = rs_from.getString(campi_from[0]); // se errore su quale
id

            for(int i=0; i < campi_from.length; i++)
            {
                out_record += campi_from[i] + "=" +
rs_from.getString(campi_from[i]) + "::" ;

                ps_into.setString(i+1, rs_from.getString(campi_from[i])); //
prepara l'insert

                in_values += campi_to[i] + "=" + rs_from.getString(campi_from[i])
+ "::" ;
            }
            out_record = out_record.substring(0, out_record.length()-2); //
tolgo il cancelletto finale

            int res = ps_into.executeUpdate();

            if(res != 0)
            {
                i_in_record++;
                in_record += i_in_record + ")" + in_values;

                // chiudi la statement
                //ps_into.close();
            }

            out_record += "\n";
            in_record += "\n";

            /*
            if(i_out_record > 10)
                break;
                */
        }

        String log = "Per la tabella origine " + tabellaFrom + " processati
in uscita record: " + i_out_record + "\n";
        log += "Per la tabella destinazione " + tabellaTo + " processati in
entrata record: " + i_in_record + "\n";
        log += "-----------------------------------------------------------
\n";
        log += out_record + in_record;

        //out.print(st);
        out.print(log);

    }
Thomas Kellerer - 31 Oct 2007 11:32 GMT
josh, 31.10.2007 11:14:
>> josh, 31.10.2007 10:22:
>>
[quoted text clipped - 77 lines]
>         //out.print(st);
>         out.print(log);

After a quick look, I'd say the following lines are your problem:

>out_record += "SELECT su record (" + i_out_record + ")";
>in_record += "INSERT su record (";
>in_record += i_in_record + ")" + in_values;

As far as I can tell, this will create huge strings with the complete
result set, as they are declared outside the loop (actually twice the
result set because one for the out_Record and one for the in_record).

Thomas
josh - 31 Oct 2007 11:48 GMT
> josh, 31.10.2007 11:14:
>
[quoted text clipped - 91 lines]
>
> Thomas

Great, the problem was not in the resultset but in my logging string!
so I by-pass
that saving storing in the DB only errors cathed...
... if I had to save, however, some logs may be I could use files ...
never string does it right?

Thanks for the help!
Daniel Pitts - 31 Oct 2007 18:03 GMT
>> josh, 31.10.2007 11:14:
>>
[quoted text clipped - 77 lines]
>
> Thanks for the help!

Specifically, look into a logging framework. log4j, commons-logging, or
even the Java Logging API.  They are designed for this type of logging.
 Log4j seems to be the most popular, and might as well be standard,
commons-logging is a facade that lets you basically plug in whatever
type of logging system you choose (such as log4j or Java Logging), and
Java Logging comes standard.

Signature

Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>

Ingo Menger - 31 Oct 2007 11:55 GMT
> josh, 31.10.2007 11:14:
>
[quoted text clipped - 89 lines]
> result set, as they are declared outside the loop (actually twice the
> result set because one for the out_Record and one for the in_record).

Actually 4-fold, since at the end he does:

              log += out_record + in_record;
               //out.print(st);
               out.print(log);

Just changing this will save 50% memory:

               out.print(out_record);
               out.print(in_record);

But, of course, the out_record and in_record logging is totally bad.
First, avoid + on strings. Use a StringBuilder instead.
If this doesn't solve the problem, write the out_record and in_record
directly to a file. For example, through a PrintWriter.
Lew - 31 Oct 2007 14:18 GMT
Thomas Kellerer wrote:
>> As far as I can tell, this will create huge strings with the complete
>> result set, as they are declared outside the loop (actually twice the
>> result set because one for the out_Record and one for the in_record).

> Actually 4-fold, since at the end he does:
>
[quoted text clipped - 3 lines]
>
> Just changing this will save 50% memory:

Actually even more than that, because all the intermediate strings are there
from each iteration.  Given n iterations, aren't there O(n^2) Strings generated?

Signature

Lew

Ingo Menger - 31 Oct 2007 14:41 GMT
> Thomas Kellerer wrote:
> >> As far as I can tell, this will create huge strings with the complete
[quoted text clipped - 10 lines]
> Actually even more than that, because all the intermediate strings are there
> from each iteration.

Sure, but the intermediates can be collected. But the "log +=
out_record + in_record" just doubles (at least) the need for
*accessible* memory in one sweep.

>  Given n iterations, aren't there O(n^2) Strings generated?

Certainly. Wich will give the gc a hard time. The program should not
only use huge amounts of memory but also many ticks in the gc.
Roedy Green - 31 Oct 2007 19:08 GMT
>in a while loop I get from a db some records and in the same time
>I put them with an insert preparedStatement in
>a destination db but the jave process increase the memory until it
>goes in out of memory.

The general advice is:

1. don't hold onto anything any longer than you have to.  The idea
would be to read one record, process one record, then let go of all
objects involved.

2. if you have to hold onto something for a while, hold onto it its
most compact form.  This may been processing it or delaying processing
it.

3. read up on packratting to make sure you are not inadvertently
holding onto something you don't need to.
http://mindprod.com/jgloss/packratting.html
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com



Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.