Java Forum / General / November 2007
memory management
josh - 31 Oct 2007 10:22 GMT Hi, I have the following problem:
in a while loop I get from a db some records and in the same time I put them with an insert preparedStatement in a destination db but the jave process increase the memory until it goes in out of memory.
How can I optimize that? also If I increase Tomcat java memory when I process more than 500 records tha java memory is already at 700mb of usage!
Please help
GArlington - 31 Oct 2007 10:46 GMT > Hi, I have the following problem: > [quoted text clipped - 8 lines] > > Please help You can loop through the records (i.e. fetch them one-by-one), reuse the same variables and run gc() after each (or few consecutive) loop iterations.
Lew - 31 Oct 2007 14:14 GMT josh wrote:
>> in a while loop I get from a db some records and in the same time >> I put them with an insert preparedStatement in [quoted text clipped - 4 lines] >> process more than 500 records >> tha java memory is already at 700mb of usage!
> You can loop through the records (i.e. fetch them one-by-one), reuse > the same variables and run gc() after each (or few consecutive) loop > iterations. It is bad advice to suggest using the gc() call.
It is also usually better to allocate new objects in the loop than to hang on to one for multiple iterations.
To the OP:
Provide an SSCCE. <http://www.physci.org/codes/sscce.html>
You have a memory leak. It is in your code. The trick is to find from where.
Without an example that recreates the error we likely cannot help in detail. Candidates are objects that are not abandoned, contrary to GArlington's suggestion. You are probably not close()ing a ResultSet or a Connection, although that usually would cause problems before OutOfMemory (OOM).
GArlington's suggestion to process records one by one is sound. Are you storing many records' worth of information at once?
You probably need to refactor some logic into separate methods and classes.
The SSCCE will show us.
 Signature Lew
Roedy Green - 31 Oct 2007 19:05 GMT >It is also usually better to allocate new objects in the loop than to hang on >to one for multiple iterations. Why is that?
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Lew - 31 Oct 2007 23:47 GMT >> It is also usually better to allocate new objects in the loop than to hang on >> to one for multiple iterations. > > Why is that? On account of the garbage collector is optimized for short-lived objects, and making long-lived ones risks having inter-generational references. In addition, the optimizer is able to optimize away the object allocation altogether in many cases. The cost of short-time object memory allocation and GC is much less than most people suppose.
I have been reading article after article explaining this and recommending that holding on to objects is not always helpful, and can even be harmful.
Also, extending an object's life artificially widens the scope of the variable, which is against good code engineering maxims.
Naturally there are situations where it's better to keep an object than reallocate, but the use of widened object scope to "optimize" turns out to be an antipattern.
Articles that shed some light on this and related topics: <http://www.ibm.com/developerworks/java/library/j-jtp01274.html?S_TACT=105AGX02&S _CMP=EDU>
<http://www.ibm.com/developerworks/java/library/j-jtp09275.html> <http://www.ibm.com/developerworks/java/library/j-jtp04223.html>
 Signature Lew
Wojtek - 05 Nov 2007 15:51 GMT Lew wrote :
> On account of the garbage collector is optimized for short-lived objects, and > making long-lived ones risks having inter-generational references. In > addition, the optimizer is able to optimize away the object allocation > altogether in many cases. The cost of short-time object memory allocation > and GC is much less than most people suppose. Sigh.
More code to refactor...
I wonder, is there a tool (Eclipse plugin?) which scans code and reduces variable scope to the least possible visibility.
 Signature Wojtek :-)
Daniel Pitts - 05 Nov 2007 19:52 GMT > Lew wrote : >> On account of the garbage collector is optimized for short-lived [quoted text clipped - 9 lines] > I wonder, is there a tool (Eclipse plugin?) which scans code and reduces > variable scope to the least possible visibility. The IDE I use will do that (IntelliJ IDEA). You can get a free trial on their website. You would want to use the Analysis tool. There may be other tools out there for this, but I don't know about them.
 Signature Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
Roedy Green - 05 Nov 2007 18:46 GMT >I have been reading article after article explaining this and recommending >that holding on to objects is not always helpful, and can even be harmful. Would this also apply to objects > 1 MB?
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Thomas Kellerer - 31 Oct 2007 10:56 GMT josh, 31.10.2007 10:22:
> Hi, I have the following problem: > [quoted text clipped - 6 lines] > process more than 500 records > tha java memory is already at 700mb of usage! My guess is that you are buffering the retrieved rows somehow (or keeping references to them). You will need to show us your code.
Thomas
josh - 31 Oct 2007 11:14 GMT > josh, 31.10.2007 10:22: > [quoted text clipped - 13 lines] > > Thomas ok I post the code:
st_from = conn.createStatement();
rs_from = st_from.executeQuery(query_from);
ps_into = conn_t.prepareStatement(insert_into);
int i_out_record = 0; int i_in_record = 0;
while(rs_from.next()) {
i_out_record++;
out_record += "SELECT su record (" + i_out_record + ")"; in_record += "INSERT su record (";
row_error = rs_from.getString(campi_from[0]); // se errore su quale id
for(int i=0; i < campi_from.length; i++) { out_record += campi_from[i] + "=" + rs_from.getString(campi_from[i]) + "::" ;
ps_into.setString(i+1, rs_from.getString(campi_from[i])); // prepara l'insert
in_values += campi_to[i] + "=" + rs_from.getString(campi_from[i]) + "::" ; } out_record = out_record.substring(0, out_record.length()-2); // tolgo il cancelletto finale
int res = ps_into.executeUpdate();
if(res != 0) { i_in_record++; in_record += i_in_record + ")" + in_values;
// chiudi la statement //ps_into.close(); }
out_record += "\n"; in_record += "\n";
/* if(i_out_record > 10) break; */ }
String log = "Per la tabella origine " + tabellaFrom + " processati in uscita record: " + i_out_record + "\n"; log += "Per la tabella destinazione " + tabellaTo + " processati in entrata record: " + i_in_record + "\n"; log += "----------------------------------------------------------- \n"; log += out_record + in_record;
//out.print(st); out.print(log);
}
Thomas Kellerer - 31 Oct 2007 11:32 GMT josh, 31.10.2007 11:14:
>> josh, 31.10.2007 10:22: >> [quoted text clipped - 77 lines] > //out.print(st); > out.print(log); After a quick look, I'd say the following lines are your problem:
>out_record += "SELECT su record (" + i_out_record + ")"; >in_record += "INSERT su record ("; >in_record += i_in_record + ")" + in_values; As far as I can tell, this will create huge strings with the complete result set, as they are declared outside the loop (actually twice the result set because one for the out_Record and one for the in_record).
Thomas
josh - 31 Oct 2007 11:48 GMT > josh, 31.10.2007 11:14: > [quoted text clipped - 91 lines] > > Thomas Great, the problem was not in the resultset but in my logging string! so I by-pass that saving storing in the DB only errors cathed... ... if I had to save, however, some logs may be I could use files ... never string does it right?
Thanks for the help!
Daniel Pitts - 31 Oct 2007 18:03 GMT >> josh, 31.10.2007 11:14: >> [quoted text clipped - 77 lines] > > Thanks for the help! Specifically, look into a logging framework. log4j, commons-logging, or even the Java Logging API. They are designed for this type of logging. Log4j seems to be the most popular, and might as well be standard, commons-logging is a facade that lets you basically plug in whatever type of logging system you choose (such as log4j or Java Logging), and Java Logging comes standard.
 Signature Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
Ingo Menger - 31 Oct 2007 11:55 GMT > josh, 31.10.2007 11:14: > [quoted text clipped - 89 lines] > result set, as they are declared outside the loop (actually twice the > result set because one for the out_Record and one for the in_record). Actually 4-fold, since at the end he does:
log += out_record + in_record; //out.print(st); out.print(log);
Just changing this will save 50% memory:
out.print(out_record); out.print(in_record);
But, of course, the out_record and in_record logging is totally bad. First, avoid + on strings. Use a StringBuilder instead. If this doesn't solve the problem, write the out_record and in_record directly to a file. For example, through a PrintWriter.
Lew - 31 Oct 2007 14:18 GMT Thomas Kellerer wrote:
>> As far as I can tell, this will create huge strings with the complete >> result set, as they are declared outside the loop (actually twice the >> result set because one for the out_Record and one for the in_record).
> Actually 4-fold, since at the end he does: > [quoted text clipped - 3 lines] > > Just changing this will save 50% memory: Actually even more than that, because all the intermediate strings are there from each iteration. Given n iterations, aren't there O(n^2) Strings generated?
 Signature Lew
Ingo Menger - 31 Oct 2007 14:41 GMT > Thomas Kellerer wrote: > >> As far as I can tell, this will create huge strings with the complete [quoted text clipped - 10 lines] > Actually even more than that, because all the intermediate strings are there > from each iteration. Sure, but the intermediates can be collected. But the "log += out_record + in_record" just doubles (at least) the need for *accessible* memory in one sweep.
> Given n iterations, aren't there O(n^2) Strings generated? Certainly. Wich will give the gc a hard time. The program should not only use huge amounts of memory but also many ticks in the gc.
Roedy Green - 31 Oct 2007 19:08 GMT >in a while loop I get from a db some records and in the same time >I put them with an insert preparedStatement in >a destination db but the jave process increase the memory until it >goes in out of memory. The general advice is:
1. don't hold onto anything any longer than you have to. The idea would be to read one record, process one record, then let go of all objects involved.
2. if you have to hold onto something for a while, hold onto it its most compact form. This may been processing it or delaying processing it.
3. read up on packratting to make sure you are not inadvertently holding onto something you don't need to. http://mindprod.com/jgloss/packratting.html
 Signature Roedy Green Canadian Mind Products The Java Glossary http://mindprod.com
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|