Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / Databases / November 2005

Tip: Looking for answers? Try searching our database.

Needed: An Efficient, reliable and smart way to copy data from one DB to another

Thread view: 
Ted Byers - 23 Nov 2005 05:20 GMT
Ok, I have a prototype running, or I should say crawling, to copy data from
an MS Access database over to an MySQL database.  I'd already set up the
tables and indeces, and was using a Java program to select all of the
contents from each table in the Access DB and insert each row from the
resulset into the MySQL.  The largest table has about 7700 rows and a couple
dozen columns,  It only takes a couple minutes to get the data from the
Access DB, but it seems to take hours to insert it into the MySQL,
undoubtedly because I've so far used the simplest approach: i.e. create a
prepared statement, and iterate through the resultset, populating the
parameters for the prepared statement and then executing it before
proceeding to the next.

I need a way to copy the data over that is much faster.  One options I can
see, but which appears to preclude using prepared statements, is to manually
parse the resultset obtained from Access to construct a really long SQL
statement (i.e. with each row consisting of comma separated values, enclosed
in parantheses, and separated from the next row by a comma).  There is a
question of how many rows ought to be inserted in one SQL statement.  A
second option suggested itself when I noticed that prepared statements have
an addBatch member function.  It wasn't clear how to use it, though, since
it doesn't appear to take any parameters, and yet the description says it
adds parameter values to a batch of SQL statements.

I do not know if either of these options I'm considering would improve
things.  Are there any other options I ought to consider?  I need to get
this done since I will need to deploy my application on a linux box.

Thanks,

Ted

Signature

R.E. (Ted) Byers, Ph.D., Ed.D.
R & D Decision Support Solutions
http://www.randddecisionsupportsolutions.com/
Healthy Living Through Informed Decision Making

Robert Klemme - 23 Nov 2005 12:53 GMT
> Ok, I have a prototype running, or I should say crawling, to copy
> data from an MS Access database over to an MySQL database.  I'd
[quoted text clipped - 20 lines]
> appear to take any parameters, and yet the description says it adds
> parameter values to a batch of SQL statements.

Just set the parameters as usual then do addBatch() instead of execute()
and call execute() only every n records.

> I do not know if either of these options I'm considering would improve
> things.  Are there any other options I ought to consider?  I need to
> get this done since I will need to deploy my application on a linux
> box.

I'd start with using batch mode of the prepared statement.  That's the
simplest change to your code, should improve things and is generally a
good idea for such an app.

Apart from that you probably first have to determine where time is spent.
Could be your network or a slow machine with the MySQL db on etc.

Kind regards

   robert
Ted Byers - 24 Nov 2005 04:54 GMT
Hi Robert,

Thanks.

> Just set the parameters as usual then do addBatch() instead of execute()
> and call execute() only every n records.
[quoted text clipped - 7 lines]
> simplest change to your code, should improve things and is generally a
> good idea for such an app.

I have done this, started the program before lunch and it is still running.

> Apart from that you probably first have to determine where time is spent.
> Could be your network or a slow machine with the MySQL db on etc.

The network isn't an issue because right now both databases are sitting on
the same machine.  And the machine is reasonably quick, with a 3.2 GHz P-IV
and 1GB of RAM.

I think Thomas might be on to something with regard to the indeces.

But I am intrigued by Hal's suggestion about exporting tables from the
Access DB through a DNS and then importing them into MySQL.  After my
program finishes, I'll give that a try.  Had I known about this option, this
task would have been done more than a week ago!  I'll spend most of tomorrow
looking for information in the MySQL documentation about how to import
existing tables into a database.

Thanks again,

Ted

Signature

R.E. (Ted) Byers, Ph.D., Ed.D.
R & D Decision Support Solutions
http://www.randddecisionsupportsolutions.com/
Healthy Living Through Informed Decision Making

Thomas Kellerer - 23 Nov 2005 16:31 GMT
Ted Byers wrote on 23.11.2005 06:20:
> Ok, I have a prototype running, or I should say crawling, to copy data from
> an MS Access database over to an MySQL database.  I'd already set up the
> tables and indeces, and was using a Java program to select all of the
> contents from each table in the Access DB and insert each row from the
> resulset into the MySQL.  The largest table has about 7700 rows and a couple

Must be something with the network. I have a program that does
essentially the same, and I can copy 10.000 records in about 30 seconds
from PostgreSQL (8.1) to MySQL (5.0) both located on the same machine.

As I don't think my program does anything magic in terms of performance,
I'd suspect that you have a network issue here.

Another thing to check is maybe the indexes you have defined on the
target table. Maybe they are slowing down inserts. Try to drop them
before copying the data, and re-create them afterwards.

Regards
Thomas
Ted Byers - 24 Nov 2005 04:45 GMT
> Ted Byers wrote on 23.11.2005 06:20:
>> Ok, I have a prototype running, or I should say crawling, to copy data
[quoted text clipped - 14 lines]
> table. Maybe they are slowing down inserts. Try to drop them before
> copying the data, and re-create them afterwards.

Hi Thomas

Thanks.  You may be on to something.  This is happening with both databases
on the same machine, one with plenty of ram and disk space, and a 3.4 GHz
P-IV.  But the database does involve  a number of indeces, including a
couple complex foreign indeces.

Cheers,

Ted

Signature

R.E. (Ted) Byers, Ph.D., Ed.D.
R & D Decision Support Solutions
http://www.randddecisionsupportsolutions.com/
Healthy Living Through Informed Decision Making

Hal Rosser - 24 Nov 2005 04:32 GMT
> Ok, I have a prototype running, or I should say crawling, to copy data from
> an MS Access database over to an MySQL database.  I'd already set up the
[quoted text clipped - 26 lines]
>
> Ted

There's an easier way -
create a DSN for your SQL database
Then right-click the table in Access and choose 'Export'
You can export the whole table easily that way.
I used this method to export tables from Access to mySQL, so I'm sure it
will also work to MS-SQL.
Ted Byers - 24 Nov 2005 04:41 GMT
> There's an easier way -
> create a DSN for your SQL database
> Then right-click the table in Access and choose 'Export'
> You can export the whole table easily that way.
> I used this method to export tables from Access to mySQL, so I'm sure it
> will also work to MS-SQL.

Um, I am using MySQL, not MS-SQL.

I already have a DSN for the Access database.  Are you suggesting creating a
DSN for the target MySQL databse also?  What is your procedure for importing
the tables into MySQL?  Is there a section about this in the MySQL
documentation that I have somehow missed?  If so, can you tell me where, in
that voluminous documentation to look.

Thanks,

Ted

Signature

R.E. (Ted) Byers, Ph.D., Ed.D.
R & D Decision Support Solutions
http://www.randddecisionsupportsolutions.com/
Healthy Living Through Informed Decision Making

Ted Byers - 24 Nov 2005 15:07 GMT
> There's an easier way -
> create a DSN for your SQL database
> Then right-click the table in Access and choose 'Export'
> You can export the whole table easily that way.
> I used this method to export tables from Access to mySQL, so I'm sure it
> will also work to MS-SQL.

Darn!  I just noticed that your method involves using MS Access, which I
don't have installed.

Oh well, back to the drawing board.

Thanks.

Ted

Signature

R.E. (Ted) Byers, Ph.D., Ed.D.
R & D Decision Support Solutions
http://www.randddecisionsupportsolutions.com/
Healthy Living Through Informed Decision Making

Ted Byers - 25 Nov 2005 19:33 GMT
> [snip]
> There's an easier way -
[quoted text clipped - 3 lines]
> I used this method to export tables from Access to mySQL, so I'm sure it
> will also work to MS-SQL.

There is a much easier way.  I visited the MySQL site today and found they
have a new migration tool.  I used it to transfer the data in a matter of
minutes.  Now I'd like to see the source code for that tool so I can see how
they did it so quckly.

Cheers,

Ted

Signature

R.E. (Ted) Byers, Ph.D., Ed.D.
R & D Decision Support Solutions
http://www.randddecisionsupportsolutions.com/
Healthy Living Through Informed Decision Making

Mark Matthews - 29 Nov 2005 18:26 GMT
>> [snip]
>> There's an easier way -
[quoted text clipped - 12 lines]
>
> Ted

Ted,

They did it using JDBC, but the migration toolkit uses "bulk" inserts of
the form:

INSERT INTO [table] VALUES (...),(...),(...)

This form of insert statement is optimized inside the MySQL server in
that it does the index update(s) all in one go for all elements of the
insert, rather than row-by-row as a simple single-row insert would do.

The migration toolkit will build a query of this size up to 5MB or so,
or the maximum sized query that your MySQL server is confiugred to accept.

My guess is that your entire table (or at least most of it) was sent as
a single statement to MySQL, which is why this method is much quicker
than the other alternatives you tried.

    -Mark
Mark Matthews - 29 Nov 2005 18:28 GMT
>> [snip]
>> There's an easier way -
[quoted text clipped - 12 lines]
>
> Ted

Ted,

As a followup, the code in question is at
http://svn.mysql.com/svnpublic/mysql-migration-tool/source/java/com/mysql/grt/mo
dules/MigrationGeneric.java


Look for the doDataBulkTransferTableToMysql() method.

Regards,

    -Mark


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.