I'm inserting about 400,000 rows into a MySQL table with
PreparedStatement. To do so, I'm inserting them in batches of 100 rows
each. At the start, they are fast (0.01 to 0.02 seconds per batch).
but after a while, they start getting slow. First, I'd see a single
batch that'd like 50 seconds in the middle of others which take just
0.02 seconds. Then, they start to average about 2-3 seconds.
I checked two explanations for this behavior. First, the table is
getting bigger so insertions are expected to get slower (there's an
index on the table). But I was able to rule this out by restarting my
program and inserting new rows into the table (without clearing it).
Inserting the new rows are fast.
Second possible cause is that other things in my program are slowing
things down. I was able to rule this out by running my program with
the exception of PreparedStatement.executeBatch() commented out. In
other words, everything except the actual insertion into the table.
This ran without any slowness.
So, I'm starting to think PreparedStatement.executeBatch() is doing
something funky. The really slow batches (50 seconds) look like it's
some sort of memory paging issue because the CPU usage is 0% during it.
Though when I observe the memory usage of the program during
execution, it doesn't seem that weird.. Another reason could be that
the table's getting locked for some weird reason during all those
insertions.
Anyway, can someone shed some light on this? It's my first time using
MySQL with Java and I'm at a loss here. Thanks!
Carl - 19 Dec 2006 06:07 GMT
Xiaolei Li 写道:
> I'm inserting about 400,000 rows into a MySQL table with
> PreparedStatement. To do so, I'm inserting them in batches of 100 rows
[quoted text clipped - 25 lines]
> Anyway, can someone shed some light on this? It's my first time using
> MySQL with Java and I'm at a loss here. Thanks!
I did the same thing two months ago. At last I gave up. I think it's not
a good idea to insert lots of data using jdbc. I use mysql's import
command instead, and it's fast!