Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / Databases / December 2006

Tip: Looking for answers? Try searching our database.

Batch insertions getting slow over time

Thread view: 
Xiaolei Li - 14 Dec 2006 05:58 GMT
I'm inserting about 400,000 rows into a MySQL table with
PreparedStatement.  To do so, I'm inserting them in batches of 100 rows
each.  At the start, they are fast (0.01 to 0.02 seconds per batch).
but after a while, they start getting slow.  First, I'd see a single
batch that'd like 50 seconds in the middle of others which take just
0.02 seconds.  Then, they start to average about 2-3 seconds.

I checked two explanations for this behavior.  First, the table is
getting bigger so insertions are expected to get slower (there's an
index on the table).  But I was able to rule this out by restarting my
program and inserting new rows into the table (without clearing it).
Inserting the new rows are fast.

Second possible cause is that other things in my program are slowing
things down.  I was able to rule this out by running my program with
the exception of PreparedStatement.executeBatch() commented out.  In
other words, everything except the actual insertion into the table.
This ran without any slowness.

So, I'm starting to think PreparedStatement.executeBatch() is doing
something funky.  The really slow batches (50 seconds) look like it's
some sort of memory paging issue because the CPU usage is 0% during it.
Though when I observe the memory usage of the program during
execution, it doesn't seem that weird..  Another reason could be that
the table's getting locked for some weird reason during all those
insertions.

Anyway, can someone shed some light on this?  It's my first time using
MySQL with Java and I'm at a loss here.  Thanks!
Carl - 19 Dec 2006 06:07 GMT
Xiaolei Li 写道:
> I'm inserting about 400,000 rows into a MySQL table with
> PreparedStatement.  To do so, I'm inserting them in batches of 100 rows
[quoted text clipped - 25 lines]
> Anyway, can someone shed some light on this?  It's my first time using
> MySQL with Java and I'm at a loss here.  Thanks!

I did the same thing two months ago. At last I gave up. I think it's not
 a good idea to insert lots of data using jdbc. I use mysql's import
command instead, and it's fast!


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.