Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / Databases / January 2006

Tip: Looking for answers? Try searching our database.

select 10000 rows from Oracle in shortest way

Thread view: 
dok - 23 Jan 2006 10:27 GMT
Looking for a good desicions for trasfering more than  10000 rows from
oracle to mysql using only jdbc. Connect is mostly slow about 1 row in
a second, rights on oracle are just for select, update, insert
operations. I think if I can devide 10000 rows in 10 parts and transfer
them in concurent threads and connections. Any suggestion can help me
to solve this problem.
Thomas Kellerer - 23 Jan 2006 11:17 GMT
> Looking for a good desicions for trasfering more than  10000 rows from
> oracle to mysql using only jdbc. Connect is mostly slow about 1 row in
> a second, rights on oracle are just for select, update, insert
> operations. I think if I can devide 10000 rows in 10 parts and transfer
> them in concurent threads and connections. Any suggestion can help me
> to solve this problem.

I can copy 10000 rows in about 40 seconds (pure JDBC).

But I have noted, that insert speed is extremely slow when autocommit is
turned on. So maybe that is your problem?

Thomas

Signature

It's not a RootKit - it's a Sony

dok - 23 Jan 2006 13:44 GMT
autocommit is a good idea, but I have slow connect with remote server
and need to select this items at max speed
HansF - 23 Jan 2006 15:09 GMT
>> Looking for a good desicions for trasfering more than  10000 rows from
>> oracle to mysql using only jdbc. Connect is mostly slow about 1 row in
[quoted text clipped - 7 lines]
> But I have noted, that insert speed is extremely slow when autocommit is
> turned on. So maybe that is your problem?

Manual commits in a loop with also slow things down.

Another frequent Java coding mistake is to build the SQL string manually
by concat'ing the literals into the string.  A knowledgeable ORacle
Java developer will use bind (aka positional) parameters when appropriate.
Under many circumstances building a SQL stateement as a string and
executing that string dynamically is a wonderful way to kill ORacle
performance.

Another thing to look at is bulk binding & array fetching..

All these techniques are well described in the online Oracle Java
Developers and JDBC Developers manuals.  Which most Java developers don't
read :-(

Signature

Hans Forbrich                          
Canada-wide Oracle training and consulting
mailto: Fuzzy.GreyBeard_at_gmail.com  
*** Top posting [replies] guarantees I won't respond. ***

Thomas Hawtin - 23 Jan 2006 16:50 GMT
> But I have noted, that insert speed is extremely slow when autocommit is
> turned on. So maybe that is your problem?

A commit would require waiting for a write to disc to complete. How slow
that is depends upon what you are logging to. Solid state memory will be
significantly faster than rotating media. When you switch auto commit
off (which should be a matter of course) make sure to commit (or
rollback) when you've finished.

If your driver supports it, then bulk updates should be very much faster
with a networked driver. If they don't perhaps you could generate an SQL
statement which takes a number of rows at once. Or perhaps moving the
code writing to the database onto the same physical machine. If you're
really, really, really desperate, write stored procedure to decode a
long argument and insert multiple rows.

Tom Hawtin
Signature

Unemployed English Java programmer
http://jroller.com/page/tackline/

Robert Klemme - 23 Jan 2006 17:20 GMT
>> But I have noted, that insert speed is extremely slow when
>> autocommit is turned on. So maybe that is your problem?
[quoted text clipped - 11 lines]
> physical machine. If you're really, really, really desperate, write
> stored procedure to decode a long argument and insert multiple rows.

Another (additional) option for more efficient resource usage could be to
use two threads (a reader and a writer thread) connected by a thread safe
queue.  That way the inserter can insert data while the reader waits for
the next packet to be sent from the server.

Yet another option might be to use SSL with compression for the slow link.
Never did this myself with a JDBC connection though.

HTH

   robert
Roedy Green - 23 Jan 2006 20:39 GMT
>Looking for a good desicions for trasfering more than  10000 rows from
>oracle to mysql using only jdbc. Connect is mostly slow about 1 row in
>a second, rights on oracle are just for select, update, insert
>operations. I think if I can devide 10000 rows in 10 parts and transfer
>them in concurent threads and connections. Any suggestion can help me
>to solve this problem.

import/export tends to be much faster than spoon feeding a row at a
time. Databases tend to turn off the transaction backout logic on
import.

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

HansF - 23 Jan 2006 21:31 GMT
> import/export tends to be much faster than spoon feeding a row at a
> time. Databases tend to turn off the transaction backout logic on
> import.

Generally agree if using the same rdbms.  But export from Oracle, import
to MySQL?
Roedy Green - 23 Jan 2006 22:43 GMT
>Generally agree if using the same rdbms.  But export from Oracle, import
>to MySQL?

The export format will likely be some flavour of CSV.  IF you are
lucky, identical. If not, you could cook up a converter using
http://mindprod.com/products1.html#CSV

The key is to avoid the commit/rollback logic on each record.

This sticks in my mind from a RFP I was involved in to purchase a new
mainframe for my employer BC Hydro.   We required  a benchmark that at
one point involved a database load. The Univac (now Unisys people)
were inexperienced and did not realise this load problem.. The
database load took about 24 hours.  The pro-IBM forces at BC Hydro
refused to let them have another shot at it,  avoiding the
commit/rollback logic.

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

HansF - 23 Jan 2006 22:52 GMT
>>Generally agree if using the same rdbms.  But export from Oracle, import
>>to MySQL?
>
> The export format will likely be some flavour of CSV.  IF you are
> lucky, identical. If not, you could cook up a converter using
> http://mindprod.com/products1.html#CSV

Ah!  

In  Oracle, the term export has a very specific meaning.  Unfortunately,
that meaning is not compatible with your usage, which could have led to
confusion.


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.