Java Forum / Databases / November 2005
Needed: An Efficient, reliable and smart way to copy data from one DB to another
Ted Byers - 23 Nov 2005 05:20 GMT Ok, I have a prototype running, or I should say crawling, to copy data from an MS Access database over to an MySQL database. I'd already set up the tables and indeces, and was using a Java program to select all of the contents from each table in the Access DB and insert each row from the resulset into the MySQL. The largest table has about 7700 rows and a couple dozen columns, It only takes a couple minutes to get the data from the Access DB, but it seems to take hours to insert it into the MySQL, undoubtedly because I've so far used the simplest approach: i.e. create a prepared statement, and iterate through the resultset, populating the parameters for the prepared statement and then executing it before proceeding to the next.
I need a way to copy the data over that is much faster. One options I can see, but which appears to preclude using prepared statements, is to manually parse the resultset obtained from Access to construct a really long SQL statement (i.e. with each row consisting of comma separated values, enclosed in parantheses, and separated from the next row by a comma). There is a question of how many rows ought to be inserted in one SQL statement. A second option suggested itself when I noticed that prepared statements have an addBatch member function. It wasn't clear how to use it, though, since it doesn't appear to take any parameters, and yet the description says it adds parameter values to a batch of SQL statements.
I do not know if either of these options I'm considering would improve things. Are there any other options I ought to consider? I need to get this done since I will need to deploy my application on a linux box.
Thanks,
Ted
 Signature R.E. (Ted) Byers, Ph.D., Ed.D. R & D Decision Support Solutions http://www.randddecisionsupportsolutions.com/ Healthy Living Through Informed Decision Making
Robert Klemme - 23 Nov 2005 12:53 GMT > Ok, I have a prototype running, or I should say crawling, to copy > data from an MS Access database over to an MySQL database. I'd [quoted text clipped - 20 lines] > appear to take any parameters, and yet the description says it adds > parameter values to a batch of SQL statements. Just set the parameters as usual then do addBatch() instead of execute() and call execute() only every n records.
> I do not know if either of these options I'm considering would improve > things. Are there any other options I ought to consider? I need to > get this done since I will need to deploy my application on a linux > box. I'd start with using batch mode of the prepared statement. That's the simplest change to your code, should improve things and is generally a good idea for such an app.
Apart from that you probably first have to determine where time is spent. Could be your network or a slow machine with the MySQL db on etc.
Kind regards
robert
Ted Byers - 24 Nov 2005 04:54 GMT Hi Robert,
Thanks.
> Just set the parameters as usual then do addBatch() instead of execute() > and call execute() only every n records. [quoted text clipped - 7 lines] > simplest change to your code, should improve things and is generally a > good idea for such an app. I have done this, started the program before lunch and it is still running.
> Apart from that you probably first have to determine where time is spent. > Could be your network or a slow machine with the MySQL db on etc. The network isn't an issue because right now both databases are sitting on the same machine. And the machine is reasonably quick, with a 3.2 GHz P-IV and 1GB of RAM.
I think Thomas might be on to something with regard to the indeces.
But I am intrigued by Hal's suggestion about exporting tables from the Access DB through a DNS and then importing them into MySQL. After my program finishes, I'll give that a try. Had I known about this option, this task would have been done more than a week ago! I'll spend most of tomorrow looking for information in the MySQL documentation about how to import existing tables into a database.
Thanks again,
Ted
 Signature R.E. (Ted) Byers, Ph.D., Ed.D. R & D Decision Support Solutions http://www.randddecisionsupportsolutions.com/ Healthy Living Through Informed Decision Making
Thomas Kellerer - 23 Nov 2005 16:31 GMT Ted Byers wrote on 23.11.2005 06:20:
> Ok, I have a prototype running, or I should say crawling, to copy data from > an MS Access database over to an MySQL database. I'd already set up the > tables and indeces, and was using a Java program to select all of the > contents from each table in the Access DB and insert each row from the > resulset into the MySQL. The largest table has about 7700 rows and a couple Must be something with the network. I have a program that does essentially the same, and I can copy 10.000 records in about 30 seconds from PostgreSQL (8.1) to MySQL (5.0) both located on the same machine.
As I don't think my program does anything magic in terms of performance, I'd suspect that you have a network issue here.
Another thing to check is maybe the indexes you have defined on the target table. Maybe they are slowing down inserts. Try to drop them before copying the data, and re-create them afterwards.
Regards Thomas
Ted Byers - 24 Nov 2005 04:45 GMT > Ted Byers wrote on 23.11.2005 06:20: >> Ok, I have a prototype running, or I should say crawling, to copy data [quoted text clipped - 14 lines] > table. Maybe they are slowing down inserts. Try to drop them before > copying the data, and re-create them afterwards. Hi Thomas
Thanks. You may be on to something. This is happening with both databases on the same machine, one with plenty of ram and disk space, and a 3.4 GHz P-IV. But the database does involve a number of indeces, including a couple complex foreign indeces.
Cheers,
Ted
 Signature R.E. (Ted) Byers, Ph.D., Ed.D. R & D Decision Support Solutions http://www.randddecisionsupportsolutions.com/ Healthy Living Through Informed Decision Making
Hal Rosser - 24 Nov 2005 04:32 GMT > Ok, I have a prototype running, or I should say crawling, to copy data from > an MS Access database over to an MySQL database. I'd already set up the [quoted text clipped - 26 lines] > > Ted There's an easier way - create a DSN for your SQL database Then right-click the table in Access and choose 'Export' You can export the whole table easily that way. I used this method to export tables from Access to mySQL, so I'm sure it will also work to MS-SQL.
Ted Byers - 24 Nov 2005 04:41 GMT > There's an easier way - > create a DSN for your SQL database > Then right-click the table in Access and choose 'Export' > You can export the whole table easily that way. > I used this method to export tables from Access to mySQL, so I'm sure it > will also work to MS-SQL. Um, I am using MySQL, not MS-SQL.
I already have a DSN for the Access database. Are you suggesting creating a DSN for the target MySQL databse also? What is your procedure for importing the tables into MySQL? Is there a section about this in the MySQL documentation that I have somehow missed? If so, can you tell me where, in that voluminous documentation to look.
Thanks,
Ted
 Signature R.E. (Ted) Byers, Ph.D., Ed.D. R & D Decision Support Solutions http://www.randddecisionsupportsolutions.com/ Healthy Living Through Informed Decision Making
Ted Byers - 24 Nov 2005 15:07 GMT > There's an easier way - > create a DSN for your SQL database > Then right-click the table in Access and choose 'Export' > You can export the whole table easily that way. > I used this method to export tables from Access to mySQL, so I'm sure it > will also work to MS-SQL. Darn! I just noticed that your method involves using MS Access, which I don't have installed.
Oh well, back to the drawing board.
Thanks.
Ted
 Signature R.E. (Ted) Byers, Ph.D., Ed.D. R & D Decision Support Solutions http://www.randddecisionsupportsolutions.com/ Healthy Living Through Informed Decision Making
Ted Byers - 25 Nov 2005 19:33 GMT > [snip] > There's an easier way - [quoted text clipped - 3 lines] > I used this method to export tables from Access to mySQL, so I'm sure it > will also work to MS-SQL. There is a much easier way. I visited the MySQL site today and found they have a new migration tool. I used it to transfer the data in a matter of minutes. Now I'd like to see the source code for that tool so I can see how they did it so quckly.
Cheers,
Ted
 Signature R.E. (Ted) Byers, Ph.D., Ed.D. R & D Decision Support Solutions http://www.randddecisionsupportsolutions.com/ Healthy Living Through Informed Decision Making
Mark Matthews - 29 Nov 2005 18:26 GMT >> [snip] >> There's an easier way - [quoted text clipped - 12 lines] > > Ted Ted,
They did it using JDBC, but the migration toolkit uses "bulk" inserts of the form:
INSERT INTO [table] VALUES (...),(...),(...)
This form of insert statement is optimized inside the MySQL server in that it does the index update(s) all in one go for all elements of the insert, rather than row-by-row as a simple single-row insert would do.
The migration toolkit will build a query of this size up to 5MB or so, or the maximum sized query that your MySQL server is confiugred to accept.
My guess is that your entire table (or at least most of it) was sent as a single statement to MySQL, which is why this method is much quicker than the other alternatives you tried.
-Mark
Mark Matthews - 29 Nov 2005 18:28 GMT >> [snip] >> There's an easier way - [quoted text clipped - 12 lines] > > Ted Ted,
As a followup, the code in question is at http://svn.mysql.com/svnpublic/mysql-migration-tool/source/java/com/mysql/grt/mo dules/MigrationGeneric.java
Look for the doDataBulkTransferTableToMysql() method.
Regards,
-Mark
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|