> Looking for a good desicions for trasfering more than 10000 rows from
> oracle to mysql using only jdbc. Connect is mostly slow about 1 row in
> a second, rights on oracle are just for select, update, insert
> operations. I think if I can devide 10000 rows in 10 parts and transfer
> them in concurent threads and connections. Any suggestion can help me
> to solve this problem.
I can copy 10000 rows in about 40 seconds (pure JDBC).
But I have noted, that insert speed is extremely slow when autocommit is
turned on. So maybe that is your problem?
Thomas

Signature
It's not a RootKit - it's a Sony
dok - 23 Jan 2006 13:44 GMT
autocommit is a good idea, but I have slow connect with remote server
and need to select this items at max speed
HansF - 23 Jan 2006 15:09 GMT
>> Looking for a good desicions for trasfering more than 10000 rows from
>> oracle to mysql using only jdbc. Connect is mostly slow about 1 row in
[quoted text clipped - 7 lines]
> But I have noted, that insert speed is extremely slow when autocommit is
> turned on. So maybe that is your problem?
Manual commits in a loop with also slow things down.
Another frequent Java coding mistake is to build the SQL string manually
by concat'ing the literals into the string. A knowledgeable ORacle
Java developer will use bind (aka positional) parameters when appropriate.
Under many circumstances building a SQL stateement as a string and
executing that string dynamically is a wonderful way to kill ORacle
performance.
Another thing to look at is bulk binding & array fetching..
All these techniques are well described in the online Oracle Java
Developers and JDBC Developers manuals. Which most Java developers don't
read :-(

Signature
Hans Forbrich
Canada-wide Oracle training and consulting
mailto: Fuzzy.GreyBeard_at_gmail.com
*** Top posting [replies] guarantees I won't respond. ***
Thomas Hawtin - 23 Jan 2006 16:50 GMT
> But I have noted, that insert speed is extremely slow when autocommit is
> turned on. So maybe that is your problem?
A commit would require waiting for a write to disc to complete. How slow
that is depends upon what you are logging to. Solid state memory will be
significantly faster than rotating media. When you switch auto commit
off (which should be a matter of course) make sure to commit (or
rollback) when you've finished.
If your driver supports it, then bulk updates should be very much faster
with a networked driver. If they don't perhaps you could generate an SQL
statement which takes a number of rows at once. Or perhaps moving the
code writing to the database onto the same physical machine. If you're
really, really, really desperate, write stored procedure to decode a
long argument and insert multiple rows.
Tom Hawtin

Signature
Unemployed English Java programmer
http://jroller.com/page/tackline/
Robert Klemme - 23 Jan 2006 17:20 GMT
>> But I have noted, that insert speed is extremely slow when
>> autocommit is turned on. So maybe that is your problem?
[quoted text clipped - 11 lines]
> physical machine. If you're really, really, really desperate, write
> stored procedure to decode a long argument and insert multiple rows.
Another (additional) option for more efficient resource usage could be to
use two threads (a reader and a writer thread) connected by a thread safe
queue. That way the inserter can insert data while the reader waits for
the next packet to be sent from the server.
Yet another option might be to use SSL with compression for the slow link.
Never did this myself with a JDBC connection though.
HTH
robert
>Looking for a good desicions for trasfering more than 10000 rows from
>oracle to mysql using only jdbc. Connect is mostly slow about 1 row in
>a second, rights on oracle are just for select, update, insert
>operations. I think if I can devide 10000 rows in 10 parts and transfer
>them in concurent threads and connections. Any suggestion can help me
>to solve this problem.
import/export tends to be much faster than spoon feeding a row at a
time. Databases tend to turn off the transaction backout logic on
import.

Signature
Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.
HansF - 23 Jan 2006 21:31 GMT
> import/export tends to be much faster than spoon feeding a row at a
> time. Databases tend to turn off the transaction backout logic on
> import.
Generally agree if using the same rdbms. But export from Oracle, import
to MySQL?
Roedy Green - 23 Jan 2006 22:43 GMT
>Generally agree if using the same rdbms. But export from Oracle, import
>to MySQL?
The export format will likely be some flavour of CSV. IF you are
lucky, identical. If not, you could cook up a converter using
http://mindprod.com/products1.html#CSV
The key is to avoid the commit/rollback logic on each record.
This sticks in my mind from a RFP I was involved in to purchase a new
mainframe for my employer BC Hydro. We required a benchmark that at
one point involved a database load. The Univac (now Unisys people)
were inexperienced and did not realise this load problem.. The
database load took about 24 hours. The pro-IBM forces at BC Hydro
refused to let them have another shot at it, avoiding the
commit/rollback logic.

Signature
Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.
HansF - 23 Jan 2006 22:52 GMT
>>Generally agree if using the same rdbms. But export from Oracle, import
>>to MySQL?
>
> The export format will likely be some flavour of CSV. IF you are
> lucky, identical. If not, you could cook up a converter using
> http://mindprod.com/products1.html#CSV
Ah!
In Oracle, the term export has a very specific meaning. Unfortunately,
that meaning is not compatible with your usage, which could have led to
confusion.