Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / Databases / February 2005

Tip: Looking for answers? Try searching our database.

Is perl better? :(((

Thread view: 
akizub@yahoo.com - 30 Jan 2005 15:34 GMT
I'm proud and certified Java programmer since 1998.
Trust me, a lot of experience! And I prefer Java.
Also I work with other languages too.
For example we support/develop big system with around 300 perl scripts.
I try to move them to Java, but...

Recently I tried port one perl script. It does nothing but creates
thousands SQLs
retrieve and store/summarize rezults. SQLs are complicated a little but
return only pair of values. And this pair is stored in hashtable.
That's all. Simple enough.

Why not to rewrite?
But in Java it runs at least twice slower!
In my understanding there is no Java/perl calculations. Store couple
values! Phoo!

All time, I assume, is wasted on Oracle 9 connection. In middleware.
The SQLs are the same, connection is the same, Oracle is the same,
machine is the same. Everything is the same. Except time. perl is
faster somehow.

Then I used prepared statements. It became worse. At least 2.5 times
slower...
What's the deal?
Any ideas why it happens?

I believe I use latest Java 1.4, JDBC (ojdbc14.jar) from Oracle as well
as latest perl DBD or Oraperl modules.
Who have better experience?

Alex Kizub.
KiLVaiDeN - 30 Jan 2005 17:42 GMT
> I'm proud and certified Java programmer since 1998.
> Trust me, a lot of experience! And I prefer Java.
[quoted text clipped - 28 lines]
>
> Alex Kizub.

Do you use Connection pool ? maybe the time involved is lost in connecting
all the time

K
akizub@yahoo.com - 30 Jan 2005 18:11 GMT
One connection. Always alive. All these hours :(((
Alex Kizub
Robert Klemme - 30 Jan 2005 18:36 GMT
> One connection. Always alive. All these hours :(((
> Alex Kizub

If the SQL is exactly the same and you used the same machine for boths tests
the most likely cause is probably your Java code.  Did you profile it?

   robert
akizub@yahoo.com - 30 Jan 2005 18:54 GMT
100%
Chris Smith - 30 Jan 2005 21:38 GMT
> 100%

Huh?  Do you mean it's using 100% of the CPU?  If so, then you
*definitely* need to aquire a profiler and figure out where that time is
being spent.  It shouldn't be taking nearly so long to issue queries.  
There are a number of mistakes you could be making, but without seeing
your code or your profiling results, there's little that we can do.

Signature

www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

akizub@yahoo.com - 30 Jan 2005 22:50 GMT
Sorry.
100% that all is the same.
100% that  Java is not wasting the time.
And I'm sure in 100% that it is something with JDBC and Java
middleware.

See my code in this tread. I posted it twice altready.
And it's so pity when I open two windows for the same Solaris and see
that every 100'th SQL comes from perl in 5 seconds when Java produce
the same answer in the same exactly time and for the same exactly SQLs
(so there is no Oracle cache or hints or statistics, tune and so on are
different) 2-3 times slower.
And, trust me. There is no more pain for me that type the same perl
command in perl windows 3 times when it is still first run for Java and
for exactly the same result total.

Loosing hope.
Alex Kizub.
Chris Smith - 31 Jan 2005 17:43 GMT
> 100% that all is the same.
> 100% that  Java is not wasting the time.
> And I'm sure in 100% that it is something with JDBC and Java
> middleware.

So, in other words, you haven't bothered to use a profiler yet.  Stop
wasting eveyone's time, then.  If you wish to struggle against the
unknown and refuse to use tools that will help you, that's your problem.  
Just go away and do it on your own.  It's becoming clear that you aren't
willing to work with anyone to solve your problem.

Signature

www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

Juha Laiho - 30 Jan 2005 19:25 GMT
akizub@yahoo.com said:
>Recently I tried port one perl script. It does nothing but creates
>thousands SQLs retrieve and store/summarize rezults. SQLs are
[quoted text clipped - 3 lines]
>Why not to rewrite?
>But in Java it runs at least twice slower!

If I recall correctly, JDBC will at some point do conversion from
the database character set to Unicode - and perl will not, unless
your environment is set up to use/require use of UTF8 character
encoding. This could be at least part of what you're seeing.

Are your programs CPU-bound? That is, are they using (almost) one second
of CPU time for each wallclock second? If so, that'd sound even more
like the problem would be in character set conversions.

Btw, is it the JDBC thin driver you're using? You might wish to compare
results when using JDBC OCI driver; there could be a difference. The
perl DBD::Oracle driver uses the OCI driver layer anyway.
Signature

Wolf  a.k.a.  Juha Laiho     Espoo, Finland
(GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
        PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

Dimitri Maziuk - 30 Jan 2005 19:45 GMT
akizub@yahoo.com sez:
...
> All time, I assume, is wasted on Oracle 9 connection. In middleware.
> The SQLs are the same, connection is the same, Oracle is the same,
[quoted text clipped - 5 lines]
> What's the deal?
> Any ideas why it happens?

Oracle's JDBC driver is very fast -- I have a Java program that
uses prepared statements to bulk-load data from text file, it's
as fast as using oracle loader.

You can reasonably expect Java to be a little slower, but not
much. A 2.5 times slowdown with prepared statements is a sure
sign that your code is broken.

Dima
Signature

Things seemed simpler before we kept computers.              -- IX, Revelation

steve - 30 Jan 2005 21:31 GMT
> akizub@yahoo.com sez:
> ...
[quoted text clipped - 17 lines]
>
> Dima

oracles jdbc is not fast, it is a piece of crap.
try using
1. the demo version from datadirect ,it is good for 15 days, but will allow
you to bench mark
2. try using the thin driver from java.
3. post your code ( minus passwords & domain), ill take a look.

steve
akizub@yahoo.com - 30 Jan 2005 22:36 GMT
>oracles jdbc is not fast, it is a piece of crap.
Looks like this for me now :(((

>1. the demo version from datadirect ,it is good for 15 days, but will allow
you to bench mark

See this is production and, actually, third party application.
Do you think they will install something new for Java when perl is
already faster?

>2. try using the thin driver from java.
Don't understand this. I tried few different JDBC thin drivers
including latest Oracle ojdbc14.jar for our particlular  Oracle9i
9.2.0.4 JDBC Drivers.

>3. post your code ( minus passwords & domain), ill take a look.
See another anwers in this thraed. Pretty simple.
Here is code. Exactly. Except the SQL.
Hashtable ht=new Hashtable();
PreparedStatement ps =
conection.getPreparedStatement("select 'key', sysdate+? from
dual");

for (int i=0; i<2000; i++)
ps.setInt(1, i);

rs = ps.executeQuery();
if (rs.next()) {
String key=rs.getString(1);
double value=0;
Double d=(Double)ht.get(key);
if (d!=null) value=d.doubleValue();
value+=rs.getDouble(2);
ht.put(key,new Double(value));

}
rs.close();
}

Still have a little hope. But less and less...
Alex Kizub.
Robert Klemme - 31 Jan 2005 10:22 GMT
> >oracles jdbc is not fast, it is a piece of crap.
> Looks like this for me now :(((
[quoted text clipped - 38 lines]
> Still have a little hope. But less and less...
> Alex Kizub.

It's quite inefficient to recreate Doubles all the time although this is
the straightforward approach.  You can easily avoid that by creating a
mutable double class and put that into the map.  This will easily be
faster.

Kind regards

   robert
akizub@yahoo.com - 31 Jan 2005 13:14 GMT
>It's quite inefficient to recreate Doubles all the time
It's not big deal. I have 2000 SQL so I have 2000 new Doubles.
Sure it's not the point to lost 10 minutes.
I'm talking about significant difference.
John C. Bollinger - 31 Jan 2005 15:52 GMT
>>It's quite inefficient to recreate Doubles all the time
>
> It's not big deal. I have 2000 SQL so I have 2000 new Doubles.
> Sure it's not the point to lost 10 minutes.
> I'm talking about significant difference.

You don't know how much of a difference it makes because you refuse to
profile.  I agree that creating 2000 Doubles ought not to take any
significant fraction of 10 minutes, but then your whole test program
ought not to take more than a few seconds unless there is a bottleneck
elsewhere (e.g. network, DB).

John Bollinger
jobollin@indiana.edu
dar7yl - 01 Feb 2005 22:08 GMT
Looking at your code, I can see immediately where your problem is.
Remember, perl syntax and java syntax are different.

> for (int i=0; i<2000; i++)
>    ps.setInt(1, i);

this loops around 2000 times, setting the same object.  time waster supreme.
Perhaps you meant to put braces {} around a larger section.

> conection.getPreparedStatement("select 'key', sysdate+? from dual");

I'm not sure that this is the correct form for what you are trying to do.
You cannot pass the name of  a column in a select clause as a parameter
to a prepared statement.

Perhaps you meant to say "select 'key', 'sysdate' from dual."
then execute your query, returning a resultset rs, which you iterate so:

   ResultSet rs = ps.executeQuery();
   while ( rs.next() )
   {
       String key=rs.getString(1 );
       double value=0;
       Double d=(Double)ht.get(key);
       if (d!=null) value=d.doubleValue();
       value+=rs.getDouble(2);
       ht.put(key,new Double(value));
   }

regards,
   Dar7yl.

> >oracles jdbc is not fast, it is a piece of crap.
> Looks like this for me now :(((
[quoted text clipped - 38 lines]
> Still have a little hope. But less and less...
> Alex Kizub.
steve - 04 Feb 2005 21:16 GMT
>> oracles jdbc is not fast, it is a piece of crap.
> Looks like this for me now :(((
[quoted text clipped - 6 lines]
> Do you think they will install something new for Java when perl is
> already faster?

no but it wil lallow you to find out the fastest you are EVER going to run.

>> 2. try using the thin driver from java.
> Don't understand this. I tried few different JDBC thin drivers
> including latest Oracle ojdbc14.jar for our particlular  Oracle9i
> 9.2.0.4 JDBC Drivers.

sorry!!
the jdbc has 2 modes.
1. pure java.
2. java callouts to c++ oci system.

it depends on how you call the java driver when you first make hte
connection.

>> 3. post your code ( minus passwords & domain), ill take a look.
> See another anwers in this thraed. Pretty simple.
[quoted text clipped - 22 lines]
> Still have a little hope. But less and less...
> Alex Kizub.

see my post else where for problems on your code.

steve
akizub@yahoo.com - 30 Jan 2005 21:40 GMT
Here is code. Exactly. Except the SQL.
Hashtable ht=new Hashtable();
PreparedStatement ps =
conection.getPreparedStatement("select 'key', sysdate+? from
dual");

for (int i=0; i<2000; i++)
ps.setInt(1, i);

rs = ps.executeQuery();
if (rs.next()) {
String key=rs.getString(1);
double value=0;
Double d=(Double)ht.get(key);
if (d!=null) value=d.doubleValue();
value+=rs.getDouble(2);
ht.put(key,new Double(value));
}
rs.close();
}

That's all. Exceptions are outside and handled once. No existing null
values.
Hashtable is 1000 or less keys. Not big deal. As well as possible
impovements of code like mutable Objects instead Double or similar.

And that's all! No Unicode expenses or huge network trafic.
Thin driver.
I'll try OCI driver. Always was sure that thin is good enough.
Keep guessing! Because it's the shock for me.

Alex Kizub.
John C. Bollinger - 31 Jan 2005 15:56 GMT
> Here is code. Exactly. Except the SQL.

Evidently it isn't.  I observe, for instance, that you misspelled
"prepareStatement" as "getPreparedStatement":

> PreparedStatement ps =
> conection.getPreparedStatement("select 'key', sysdate+? from
> dual");

There is at least one other likely error in your code that has already
pointed out elsewhere (the extent of your loop), but I don't know
whether that's in the real code or whether it's a typo.

John Bollinger
jobollin@indiana.edu
steve - 04 Feb 2005 21:13 GMT
> Here is code. Exactly. Except the SQL.
> Hashtable ht=new Hashtable();
[quoted text clipped - 28 lines]
>
> Alex Kizub.

loose the first "if" & replace with a while.
loose the checks for types inside you loop, replace it with a  getobject.

that should get you as fast as you are going to be using the thin driver.,
using the above code.

but you need to rewrite the whole thing, why loop round a prepared statement
2000 times, that is just madness.

try the following system :

write the code as a serverside pl/sql routine that returns a cursor of 2,000
cols, then call the routine.
it will reduce your network calls by atleast 50%, and reduce the servers sh.t 
down even further.

but i cannot see why you have to do what you are doing , the way you do it.

steve
Chris Smith - 30 Jan 2005 21:42 GMT
> You can reasonably expect Java to be a little slower, but not
> much.

I don't know about that; a little slower than C or FORTRAN in the
average case, sure.  A little slower than Perl, though?  Not unless you
tune the test specifically to exploit the strengths of Perl and
weaknesses of Java; and even then I wouldn't guarantee you could produce
these results easily.

In any case, something is clearly wrong here besides the language.

Signature

www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

Dimitri Maziuk - 31 Jan 2005 17:11 GMT
Chris Smith sez:
>> You can reasonably expect Java to be a little slower, but not
>> much.
>
> I don't know about that; a little slower than C or FORTRAN in the
> average case, sure.  A little slower than Perl, though?

JVM startup time and garbage collection.

> In any case, something is clearly wrong here besides the language.

Yep. I'd look into CLOSE_CURSORS_ON_COMMIT (or whatever it's
called) if I were OP.

Dima
Signature

... If you want to make sure you don't put a Pig in a List of airplanes and
have it fail at insertion rather than extraction, use
planelist.add((Airplane)o) instead of planelist.add(o).  It's that easy.
                                                    -- Mark 'Kamikaze' Hughes

Tor Iver Wilhelmsen - 31 Jan 2005 19:15 GMT
> JVM startup time and garbage collection.

Perl has (in theory at least) both of those as well, though.
Luke Webber - 31 Jan 2005 21:21 GMT
>>JVM startup time and garbage collection.
>
> Perl has (in theory at least) both of those as well, though.

And we're talking about a process that runs for as much as ten minutes,
so it should not be significant.

Luke
Walter Mitty - 07 Feb 2005 12:54 GMT
>>> JVM startup time and garbage collection.
>>
>> Perl has (in theory at least) both of those as well, though.
>
> And we're talking about a process that runs for as much as ten minutes,
> so it should not be significant.

Since he's creating/deleting all the time (probably), it might well be.
Tom Dyess - 30 Jan 2005 21:20 GMT
> I'm proud and certified Java programmer since 1998.
> Trust me, a lot of experience! And I prefer Java.
[quoted text clipped - 28 lines]
>
> Alex Kizub.

Yes, I would have to take a look at the code of the Java application. Was
the original PERL script hitting an Oracle database? Why are you calling
thousands of SQL statements that just return a pairs? Is there any way to
call several of these statements in a single call? PERL is great for text
manipulation, but I don't think this fact would cover the gammut of ~2.5x
difference in speed. Did you run an explain plan on your Oracle SQL? Is your
Oracle box the same DB that the perl scripts are calling? (Ie perl hitting
production, java hitting some crappy 90mhz oracle box someone threw
together). Are you using SQL hints when tuning your SQL? When you step
through a debug, are the SQL calls taking the most time? Are you using the
same Oracle client software on both test scenarios? Are you using a
persistant connection? (ie not dropping and reconnecting every script). Are
you creating a Java application for every script, or are you creating a
large single Java app that calls all scripts?

I don't want the answers to all these, just suggesting some things to try. I
know you said you were using the same Oracle, but if all you are doing is
sending a SQL statement to Oracle, Java is done until you get the result. It
doesn't sound the same. By same I mean you are connecting to the same
server, same instance with the same client on the same workstation, just two
different languages. Does your workstation have enough free memory to be
running Java?

On OraclePower.com (ya ya, shameless plug, but I *do* have a point) I run 17
queries on the home page, and it comes back instantly. Oracle hauls-a.s with
Java.

Signature

Tom Dyess
OraclePower.com

akizub@yahoo.com - 30 Jan 2005 22:28 GMT
Tom:
Thanks for sharing you experience. I really appreciate it.
Especially these are question which I would ask too.
As you already read I have big experience too.

> Wasthe original PERL script hitting an Oracle database?
Exactly the same.

> Why are you calling thousands of SQL statements that just return a
pairs?
I hate such design too. Let say in another query from different
database (and different time) we have these pairs.

>Is there any way to call several of these statements in a single call?
No. pairs are different.

> PERL is great for text manipulation, but I don't think this fact
would cover the gammut of ~2.5x
difference in speed.

That was my impression too. Just before yetaerday's night.

> Did you run an explain plan on your Oracle SQL?
I know I can improve this query. But that's not the point. It will be
improved in both languages and it still will be no difference. SQLs are
exactly the same.
Stupid, impractical but the same.

> Is your Oracle box the same DB that the perl scripts are calling? (Ie
perl hitting
production, java hitting some crappy 90mhz oracle box someone threw
together).

Oracle on one box, Java and perl on another. Both production.

> Are you using SQL hints when tuning your SQL?
Yes and no. The same result. See previous answer about SQL.

> When you step through a debug, are the SQL calls taking the most
time?
Don't use debug only start and end time. Code is simple. See my another
answer in the thread.

>Are you using the same Oracle client software on both test scenarios?
Yes.

> Are you using a persistant connection? (ie not dropping and
reconnecting every script).
Yes. One connection and it's alive for hours. No bad connections or
another exceptions.
Results are the same in perl and Java. But perl runs 2.5 faster.
I run two windows on the same Solaris box. In one I run java. In
another in the same time (it's about one hour) I can run perl program
twice for the same result and have another 10-15 minutes for third run.

> Are you creating a Java application for every script, or are you
creating a
large single Java app that calls all scripts?
All SQLs are in one thread and in one cycle. One by one. Just like
perl.
Of course creating of thousand of threads could be more useful. But
what about poor Oracle? To be honest, our DBAs already called me and
asked to stop my SQL and run in the middle of the night. It was first
call from them in 5 years.

>I don't want the answers to all these, just suggesting some things to try.
I tried these and a lot of others.
BECAUSE I DON"t WANT TO USE PERL!
I like it too, but support of hundreds of scripts is a nightmare!
And what arguments do I have to switch to Java?
20 minutes vs. hour!

>I know you said you were using the same Oracle, but if all you are
doing is
sending a SQL statement to Oracle, Java is done until you get the
result. It
doesn't sound the same. By same I mean you are connecting to the same
server, same instance with the same client on the same workstation,
just two different languages.

Don't use threads. As well as perl. See previous answer.

>Does your workstation have enough free memory to be running Java?
This is Solaris with 8Gb. Sure enough to store 1000 keys Hashtable.

>On OraclePower.com (ya ya, shameless plug, but I *do* have a point) I
run 17 queries on the home page, and it comes back instantly. Oracle
hauls-a.s with Java.

Sorry, don't understand. A lot of resources. Don't know how they can
help me. Mostly I know all this and always was sure that Oracale loves
Java as well as me. But it doesn't help me to prove my point for
managers.
Actually what point? Fact is fact. I can't deny it even I don't like
it.

Maybe it's OCI. But I can't install it on produstion server. I should
use what already exists. So, perl has OCI, JDBC doesn't and perl
forever?
Very sad Certified and with huge experience Java programmer
Alex Kizub.
Tom Dyess - 30 Jan 2005 22:58 GMT
> Tom:
> Thanks for sharing you experience. I really appreciate it.
[quoted text clipped - 98 lines]
> Very sad Certified and with huge experience Java programmer
> Alex Kizub.

The point with the Oraclepower comment was that it is written in Java and
runs very fast with Oracle. It was also a shameless plug, so you can ignore
it. Lol. I'm using oraclejdbc14.jar and connect using this string:

jdbc:oracle:thin:" + username + "/" + password + "@" + server + ":" + port +
":" + sid;

using this driver:

oracle.jdbc.driver.OracleDriver

I would suggest stepping through your code and see if there is any
distinguishable pause between the line before the execution and after. This
may help to determine if its the connection or the post-processing that is
making the difference. I would guess the latter, but it's better to not
guess at this point.

I'm gonna see if I can optimize the code a little, but I'm relatively new to
Java compared to Delphi, so I don't know that I will be able to do much.

Tom Dyess
OraclePower.com
Tom Dyess - 30 Jan 2005 23:19 GMT
>> Tom:
>> Thanks for sharing you experience. I really appreciate it.
[quoted text clipped - 122 lines]
> Tom Dyess
> OraclePower.com

Ok, after looking at the code, I see some mistakes, so would you send the
exact code including the SQL statement? For example

   for (int i=0; i<2000; i++)
     ps.setInt(1, i);

   rs = ps.executeQuery();

I think you mean to put the rs = ps.executeQuery() in a block when it
currently isn't:

   for (int i=0; i<2000; i++) {
     ps.setInt(1, i);

     rs = ps.executeQuery();
   }

Please send the exact code so we can run it and establish a time baseline
before tuning. Also,

   connection.getPreparedStatement("select 'key', sysdate+? from dual");

If you are using java.sql.Connection, it's prepareStatement(String), but I
don't know if you have the java.sql.Connection wrapped in an aggregating
object or not (I do).

Signature

Tom Dyess
OraclePower.com

kjc - 31 Jan 2005 06:08 GMT
>>>Tom:
>>>Thanks for sharing you experience. I really appreciate it.
[quoted text clipped - 163 lines]
> don't know if you have the java.sql.Connection wrapped in an aggregating
> object or not (I do).

Dude, I think you badly want to retract your comments.
The code posted by the OP is correct along one dimension.
He is setting a query value 2K times FIRST, THEN, issuing the query.

Your code example sends a query 2K times to the DB server.
Defintely, NOT a good idea.
Tom Dyess - 31 Jan 2005 13:58 GMT
>>>>Tom:
>>>>Thanks for sharing you experience. I really appreciate it.
[quoted text clipped - 170 lines]
> Your code example sends a query 2K times to the DB server.
> Defintely, NOT a good idea.

Dude! Thanks for that memo from the department of the obvious. Lol. I'm
trying to determine which he wants. Trying to understand the idea behind
what he is trying to do without actually seeing the full code. Based on his
comments, hitting the SQL 2000 times seems like that's what he wants to do,
which is yes, a bad idea, but if that's a requirement, that's a requirement.
In that case, NO I don't want to retract the statement, because to
troubleshoot, you need to completely understand a) what's currently
happening, b) what the desired effect is and in this case c) how to achieve
the desired effect in a much faster manner.

Consider this snippit: Tell me, what's the point of setting a prepared
statement 2000 times without executing the statement? Does that seem strange
to you? It does to me. When I see that and not the proper factory creation
for a prepared statement, connection.prepareStatement(String), it sends a
red flag to which I then presented that reply for further clarification. I
wasn't suggesting running the query 2000 times to increase the speed, I was
merely asking if that is what he wanted based on the code that I saw. KJC,
please read the entire thread first before blurting out. Alex, please
clarify.

conection.getPreparedStatement("select 'key', sysdate+? from
dual");

for (int i=0; i<2000; i++)
   ps.setInt(1, i);

rs = ps.executeQuery();

Tom Dyess
OraclePower.com
akizub@yahoo.com - 31 Jan 2005 14:30 GMT
Sorry. Of course I mistyped this code and forgot one { after for.
I think it's obvious because I had odd number of braces and there is no
sense to set parameter 2000 times and then do one execution.

OK. Here is "exact" (almost) code.
Point is not in few new Double or SQL tune.
It is exactly the same in perl.
Point is significant difference in time.
Java is about 2.5 slower!
Probably this is OCI Orcale connection for perl.
I can't install it for JDBC (don't have rights).

If I use more simple SQL (something like select sysdate from dual)
then perl and Java have the same time.
Which is obvious because bottle neck is Oracle.
But even in this example Oracle is main time eater!
And why Java realization is slower I have no idea.

//================ DA.java ===============
import java.sql.*;
import java.util.*;
import java.io.*;
import java.net.*;

public class DA {
String url;
String user;
String password;
public Connection con;
public Statement stmt;

/** Load Oracle driver one time for application */
static {
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
} catch (Exception ex) {
System.out.println("static driver not found: " + ex.getMessage());
}
}

public DA () {
}

public String openCon(String url, String user, String password) {
if (url instanceof String
&& url.length() > 0
&& user instanceof String
&& user.length() > 0
&& password instanceof String
&& password.length() > 0) {
this.url = url;
this.user = user;
this.password = password;
try {
con = DriverManager.getConnection(url, user, password);
if (con == null)
return "ERROR: DBAccess: connection is null!";
stmt = con.createStatement();
if (stmt == null)
return "ERROR: DBAccess: statement is null!";
} catch (Exception ex) {
System.out.println(
"Connection failed "
+ ex.getMessage()
+ "\n"
+ url
+ "\n"
+ user
+ " "
+ password);
return "Connection failed " + ex.getMessage() + "\n" + url + "\n";
}
return "OK";
} else
return "Error.";
}

/** Close current database connectionand statements */
public String closeCon() {
try {
stmt.close();
con.close();
return "OK";
} catch (Exception ex) {
return ("close connection " + ex.getMessage());
}

}

public static void main(String[] args) {
System.out.println("DBAccess test START!");

new DA1().run();

System.out.println("DBAccess test END!
Java="+System.getProperty("java.version","Not defined"));

}

public void run() {
openCon("jdbc:oracle:thin:@yahoo.com:1521:users",
"scott","tiger");

try {
ResultSet rs = null;
PreparedStatement ps =
getPreparedStatement("select sysdate+? from dual");
ps.setInt(1, 365);

rs = ps.executeQuery();
if (rs.next()) {
System.out.println("Prepared statement " + rs.getString(1));
}
rs.close();

} catch (Exception ex) {
System.out.println("SQL run Exception " + ex.getMessage());
}

closeCon();
}

public PreparedStatement getPreparedStatement(String sql)
throws java.sql.SQLException {
return con.prepareStatement(sql);
}

//////////// method itself ///////////////
public Vector table; // vector of String[5]; 2000 elements
// collecting code, month, userid, start, end,
public Hashtable totaled_hrs=new Hashtable();
public void getData()
{
try{
DA da=new DA();
ResultSet rs;
Properties pr=new Properties();
pr.load("/config/common.properties");

da.url=pr.getProperty("URL");
da.user=pr.getProperty("User");
da.password=pr.getProperty("Password");

if (!"OK".equals(da.openCon(da.url,da.user,da.password)))
System.out.println( "Error1");

double value=0;
String tmp[]=null;
double total=0,time=0;
String month="00";

String sql="SELECT SUM(DECODE(TO_CHAR(e.date, 'MM'), ?, e.hours,
0)) hours, \n"+
"d.person_type \n" +
"FROM projects a, tasks b, assignments c, resources d, hours e,
\n"+
"super_projects h, users g  \n"+
"WHERE \n"+
"d.id = ? \n"+
"AND (e.date >= TO_DATE(?, 'mm/dd/yyyy')) \n"+
"AND (e.date <= TO_DATE(?, 'mm/dd/yyyy')) \n"+
"AND a.id = b.projectid \n"+
"AND b.id = c.taskid \n"+
"AND c.resourceid = d.id \n"+
"AND c.id = e.assignmentid \n"+
"AND e.approved = 'yes' \n"+
"AND a.superid = h.id \n"+
"AND h.name like 'Billed %' \n"+
"AND a.name not like 'Local %' \n"+
"GROUP BY d.person_type \n";

PreparedStatement ps = da.getPreparedStatement(sql);

for(int i=0; i<table.size(); i++)
{
tmp=(String[])(table.get(i));

ps.setString(1, tmp[1]);
ps.setString(2, tmp[2]);
ps.setString(3, tmp[3]);
ps.setString(4, tmp[4]);
rs = ps.executeQuery();

if (rs.next())
{
value=rs.getDouble(1);

if ("volunteer".equals(rs.getString(2)))
{
time=0;
}
else
{
time=value;
}
}
total+=time;
Double d=(Double)totaled_hrs.get(tmp[0]);
if (d!=null)
{
time += d.doubleValue();
}
totaled_hrs.put(tmp[0], new Double(adjusted_summed_time));

}
rs.close();
}
da.closeCon();
System.out.println("Hours: "+total);
} catch (Exception e){
System.out.println( "Error "+e); }

}

}
################# perl ########################
#!/perl/5.6/bin/perl
use lib "/perl/5.6/DBD/oracle8.1.6/lib/site_perl/5.6.0/sun4-solaris";
use Oraperl;

... open connection ....

sub getData{

my(@persons)=@_;
$total=0;
%totaled_hrs=();

foreach $key(@persons){

@one_person=split(/:/,$key);

$sql=...;

$db_list=&ora_open($lda,$sql) || die "\nCan't open cursor
1: $ora_errstr\n";

($time, $type)=&ora_fetch($db_list);

if($type eq "volunteer"){
$time=0;
}
$totaled_hrs{$one_person[0]}+=$time;
$total+=$time;

}

Alex Kizub.
Robert Klemme - 31 Jan 2005 16:14 GMT
> Sorry. Of course I mistyped this code and forgot one { after for.
> I think it's obvious because I had odd number of braces and there is no
> sense to set parameter 2000 times and then do one execution.
>
> OK. Here is "exact" (almost) code.

Apparently not.  I suggest you post code that compiles and is properly
indented.  If you want feedback from people around you have to do at least
a bit to make people's life's simpler.

Regards

   robert
akizub@yahoo.com - 31 Jan 2005 16:45 GMT
>Apparently not.  I suggest you post code that compiles and is properly
indented.

This code works.
Unfortunately, when it was posted to the news/forum it was broken.
But you can see the idea. And see critical parts. And see that
everything is ok.

When I post short cut from code people want to see whole. When I post
whole code people don't like because it's too big.
Why do you suspect me that I miss double quote or curly braces and
start this big discussion?

Code is simple in both languages.
And this is not about
- tune SQL
- make connection
- improve performance
- change hardware, software, database or it's structure or algorithm
- do my work

Question is simple:
Why for the same SQL, hardware, database, network, time and so on
for small amount of 2000 SQL perl does it job for 17 minutes when java
does for 43?

If you divide these 43 miniutes for 2000 you can see that bottle neck
is definitely not couple new Doubles but connection to Oracle. And as
you can see SQL are too slow.
But why they are faster for perl? There is only one connection, as you
can see in source. What slows SQLs so much?

Agian. I don't like this database. It s... I don't like this SQL. I
don't like this project which was created ASAP (As Stupid As Possible)
and so on so on.

But  qestion remains: why it is so significant difference for two
languages for all the same environment/etc...?

Alex Kizub.
kjc - 31 Jan 2005 16:56 GMT
>>Apparently not.  I suggest you post code that compiles and is properly
>
[quoted text clipped - 37 lines]
>
> Alex Kizub.

Alex,
the best thing to do is run it through a profiler.
After revewing the output from a profiler, you should have your answer.
Luke Webber - 31 Jan 2005 21:43 GMT
>>Apparently not.  I suggest you post code that compiles and is properly
>
[quoted text clipped - 4 lines]
> But you can see the idea. And see critical parts. And see that
> everything is ok.

Nope, not a bit. I've gotta tell tou, that code isn't even close to
running. What are all these references to DA1? Where is "table"
initialised (hint: it's not). Also, you have too many right-braces in
your getData method before the catch statement, "adjusted_summed_time"
is not defined, the call to pr.load(InputStream) is passing a filename
instead of an InputStream...

> When I post short cut from code people want to see whole. When I post
> whole code people don't like because it's too big.
> Why do you suspect me that I miss double quote or curly braces and
> start this big discussion?

Calm down. It's not suspicion, we're trying to help, but if you want us
to help, you need to be clear on what you're trying to do. And you've
been very unclear to date.

> Code is simple in both languages.

Actually, the PERL code is a lot simpler. I don't speak PERL myself, but
if that's all there is to the PERL, you're doing a lot more in the Java
version.

> And this is not about
> - tune SQL
[quoted text clipped - 7 lines]
> for small amount of 2000 SQL perl does it job for 17 minutes when java
> does for 43?

Show us working code and maybe we can tell you. Perhaps you could put it
up on a web server somewhere for us to download. And /please/ make sure
that it's complete, or you're wasting the time of people who are trying
to help you. For free, I might add.

Luke
akizub@yahoo.com - 01 Feb 2005 04:31 GMT
Luke:
>we're trying to help
And I really appreciate it.
I'm only a little bit upset when people say that I'm unwiling to
work...

Let's start again with willings.

Profile doesn't have sense for me. It's obvious.
We are waiting for Oracle answer:
java -Xrunhprof:cpu=samples -classpath .:ojdbc14.jar DA
rank   self  accum   count trace method
1 74.08% 74.08%     889   102 java.net.SocketInputStream.socketRead0
2  5.42% 79.50%      65   133
java.net.SocketOutputStream.socketWrite0
3  1.33% 80.83%      16   145 oracle.jdbc.ttc7.TTCItem.unmarshal
4  1.17% 82.00%      14     4 java.lang.ClassLoader.defineClass0
5  0.92% 82.92%      11    19
java.lang.ClassLoader.findBootstrapClass
6  0.83% 83.75%      10   143
oracle.jdbc.ttc7.MAREngine.unmarshalCLRforREFS
7  0.58% 84.33%       7   159
oracle.jdbc.ttc7.TTCAdapter.createNonPlsqlTTCColumnArray
8  0.58% 84.92%       7   150
oracle.jdbc.ttc7.TTC7Protocol.createDBItem
9  0.50% 85.42%       6   158 java.lang.StringBuffer.toString

I do not believe in profiling. It takes time.
And during this time Oracle is working too!
So, I prefer more simple way - just count milliseconds.
For whole program, for database, (for sql), for cycle.
See code.

And here is exact code (except sql):
//========= DA.java =============

import java.io.*;
import java.sql.*;
import java.util.*;

public class DA {
public Connection con;
public Statement stmt;

static {
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
} catch (Exception ex) {
System.out.println("static driver not found: "
+ ex.getMessage());
}
}

public void openCon(String url,
String user, String password) {
try {
con = DriverManager.getConnection(url, user, password);
stmt = con.createStatement();
} catch (Exception ex) {
System.out.println(ex);
}
}

public void closeCon() {
try {
stmt.close();
con.close();
} catch (Exception ex) {
System.out.println(ex);
}
}

public static void main(String[] args) {
new DA().getData();
System.out.println("Java="+System.getProperty("java.version"));
}

public PreparedStatement getPreparedStatement(String sql)
throws java.sql.SQLException {
return con.prepareStatement(sql);
}

//////////// method itself ///////////////
public Vector table; // vector of String[4]; 2000 elements
// collecting_code|userid|start|end
public Hashtable totaled_hrs=new Hashtable();
public void getData()
{
try{

double mth_hours=23*7.75;  // for December 2004
String oldCode=null;

table=new Vector();
String tmp[]=null;
BufferedReader br=new BufferedReader(new FileReader("load"));
String line;
while ((line=br.readLine())!=null){
StringTokenizer st=new StringTokenizer(line,"|");
tmp=new String[4];
tmp[0]=st.nextToken();
tmp[1]=st.nextToken();
tmp[2]=st.nextToken();
tmp[3]=st.nextToken();
table.add(tmp);
}
br.close();
System.out.println(table.size());
long tall,tdb,tsql,tcycle,v2,v3;
tall=tdb=tsql=tcycle=0;

DA da=new DA();
ResultSet rs;
Properties pr=new Properties();
pr.load(new FileInputStream("common.properties"));

//openCon("jdbc:oracle:thin:@yahoo.com:1521:users",
//        "scott","tiger");

da.openCon(pr.getProperty("url"),
pr.getProperty("user"),pr.getProperty("password"));

double value=0,total=0;
String type="";

String sql=
"SELECT SUM(DECODE(TO_CHAR(e.date, 'MM'), \n"+
" '12', e.hours,0)) hours, \n"+
" d.person_type \n" +
" FROM projects a, tasks b, assignments c, \n"+
" resources d, hours e, \n"+
" super_projects h, users g  \n"+
" WHERE \n"+
" d.id = ? \n"+
" AND (e.date >= TO_DATE(?, 'mm/dd/yyyy')) \n"+
" AND (e.date <= TO_DATE(?, 'mm/dd/yyyy')) \n"+
" AND a.id = b.projectid \n"+
" AND b.id = c.taskid \n"+
" AND c.resourceid = d.id \n"+
" AND c.id = e.assignmentid \n"+
" AND e.approved = 'yes' \n"+
" AND a.superid = h.id \n"+
" AND h.name like 'Billed %' \n"+
" AND a.name not like 'Local %' \n"+
" GROUP BY d.person_type \n";

PreparedStatement ps = da.getPreparedStatement(sql);
tall=System.currentTimeMillis();

v2=System.currentTimeMillis();

for(int i=0; i<table.size(); i++)
{
tmp=(String[])(table.get(i));
v3=System.currentTimeMillis();
tcycle+=v3-v2;

ps.setString(1, tmp[1]);
ps.setString(2, tmp[2]);
ps.setString(3, tmp[3]);
rs = ps.executeQuery();
tsql+=System.currentTimeMillis()-v3;
if (!rs.next()) continue;
value=rs.getDouble(1);
type=rs.getString(2);
rs.close();

v2=System.currentTimeMillis();
tdb += v2-v3;

if (value>mth_hours&&"Billed".equals(type)) value=mth_hours;

total+=value;
MyDouble d=(MyDouble)totaled_hrs.get(tmp[0]);
if (d!=null) d.value+=value;
else {
d=new MyDouble();
d.value=value;
totaled_hrs.put(tmp[0],d);
}
if (!(tmp[0].equals(oldCode))&&oldCode!=null)
System.out.println(
oldCode+" "+
((MyDouble)totaled_hrs.get(oldCode)).value+
" "+total);
oldCode=tmp[0];
}// end cycle

da.closeCon();
tall=System.currentTimeMillis()-tall;

System.out.println("Hours: "+total+
" time all="+tall+
" time db="+tdb+
" (sql="+tsql+")"+
" time cycle="+tcycle);
} catch (Exception e){
e.printStackTrace();
}
}

class MyDouble{public double value=0;}

}
//////////// end DA.java /////////////

I hope it will post well. I'll double check it and
if it will be bad I'll put it somewhere in my web site.

And here is perl code:
################# perl ########################
#!/perl/5.6/bin/perl
use lib "/perl/5.6/DBD/oracle8.1.6/lib/site_perl/5.6.0/sun4-solaris";
use Oraperl;

... open connection ....

sub getData{

my(@persons)=@_;
$total=0;
%totaled_hrs=();

foreach $key(@persons){

@one_person=split(/:/,$key);

$sql=...;

$db_list=&ora_open($lda,$sql) || die "\nCan't open cursor
1: $ora_errstr\n";

($time, $type)=&ora_fetch($db_list);

if ($time>178.25) {
if ( "Billed" eq $type ) {
$time=178.25;
}}

$totaled_hrs{$one_person[0]}+=$time;
$total+=$time;

}
############# end of perl ################

Result of java is
Hours: 154930.95 time all=5436908 time db=5297897 (sql=5434009) time
cycle=243005
Java=1.4.2_01

Which means 90 minutes for whole run.
Where 88 minutes for Oracle waiting.
Total Java expenses 4.5 minutes.
(Please, don't be too strict to numbers.
I just cut&paste them. Milliseconds are a little big time period.
Be indulgent to this quick and not precision way.)

perl result is:
Hours: 154930.95 for 1487 seconds.
Which is 25 minutes today.
Which is 3.6 times faster.

So, I'm open to any suggestions how to improve my Java code
or how to force managers to use Java in this project.

Alex Kizub.
bugbear - 01 Feb 2005 10:50 GMT
> I do not believe in profiling. It takes time.

You ask experts for advice, and then refuse
to take it. Your choice.

   BugBear
akizub@yahoo.com - 01 Feb 2005 13:09 GMT
> > I do not believe in profiling. It takes time.

>You ask experts for advice, and then refuse to take it. Your choice.

>    BugBear
Read the post! I did profile! And it was as I expected.
Program itself is so simple that do not require profile.
It's obvious that problem is somewhere in Oracle connection.
Which means that it doesn't matter how good is your program -
connection kills it.
Very pity.

I gave up and left this application on perl.

Alex Kizub.
Luke Webber - 01 Feb 2005 21:49 GMT
>>>I do not believe in profiling. It takes time.
>
[quoted text clipped - 10 lines]
>
> I gave up and left this application on perl.

It seems clear that you're correct about the time being spent in the
JDBC connection logic, but I'm wondering just which part of the JDBC
code is taking all the time. The figures aren't fine-grained enough.

I have one off-the-wall suggestion. Why not try removing the
"rs.close()" statement and see how it goes? I have a niggling feeling
that it might be making an extra round trip to the database server for
some reason.

Cheers,
Luke
Thomas Kellerer - 01 Feb 2005 22:09 GMT
Luke Webber wrote on 01.02.2005 22:49:
> I have one off-the-wall suggestion. Why not try removing the
> "rs.close()" statement and see how it goes? I have a niggling feeling
> that it might be making an extra round trip to the database server for
> some reason.

I don't think that's a good idea. That will leave him with a lot of
unclosed cursors...

Thomas
Robert Sundström - 02 Feb 2005 16:37 GMT
> Luke Webber wrote on 01.02.2005 22:49:
>> I have one off-the-wall suggestion. Why not try removing the
[quoted text clipped - 4 lines]
> I don't think that's a good idea. That will leave him with a lot of
> unclosed cursors...

Not necessarily.

If I understand things, the original poster has an iteration where he
executes a PreparedStatement over and over again. Any open ResultSet
objects originating from that PreparedStatement should, if the JDBC driver
complies with the JDBC spec, be automatically closed at the next execute.

However, it would be most strange if the 2000 extra round trips to the
server can cause the delay. I would say that the server, for some reason
optimizes the SQL-statements differently. It could be that the Oracle-Perl
driver internally flips some magic option or compilation directive which
in turn makes the optimizer make better or less good decisions. The
original poster is probably better off asking this question at an Oracle
group.
Signature

Robert Sundström, Mimer SQL Development
Mimer Information Technology AB, http://www.mimer.com
Validate your SQL statements/procedures at
http://developer.mimer.com/validator

Dimitri Maziuk - 02 Feb 2005 18:45 GMT
Thomas Kellerer sez:
> Luke Webber wrote on 01.02.2005 22:49:
>> I have one off-the-wall suggestion. Why not try removing the
[quoted text clipped - 4 lines]
> I don't think that's a good idea. That will leave him with a lot of
> unclosed cursors...

Nope. Oracle doesn't close the cursor on rs.close(), it closes on
query.close() (by default, anyway).

Dima
Signature

The wombat is a mixture of chalk and clay used for respiration.      -- MegaHal

steve - 04 Feb 2005 21:34 GMT
>>>> I do not believe in profiling. It takes time.
>>
[quoted text clipped - 22 lines]
> Cheers,
> Luke

he should not be calling an sql statement 2000 times, when he can do it in
the server, and return a cursor.

and if he is so stubbon to do a callout 2000 times he can bulk it up to 1
call, execute & then close.
not callout 2,000 times , closing each time.

the code is broken, implementation wise, and it can be profiled to the end of
time,  but it is not going to be able to fix it  like that.

steve
Chris Smith - 05 Feb 2005 03:18 GMT
> he should not be calling an sql statement 2000 times, when he can do it in
> the server, and return a cursor.

Clearly this is only true in a limited set of situations.  Neither you
nor I know enough about Alex's application to know if it's true or not.  
Nevertheless, it's not at all relevant to the question of why the
performance between Java and Perl might differ.

It appears the point of your post is that you feel smart.  I'm glad you
feel smart.  Self-esteem is always a good thing.  When you follow up
with incorrect advice, it makes you seem even smarter.  When you
accentuate your self-esteem with random cursing in public to show your
disdain for others, it makes you seem like a really nice guy.

Signature

www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

steve - 04 Feb 2005 21:38 GMT
>>>> I do not believe in profiling. It takes time.
>>
[quoted text clipped - 22 lines]
> Cheers,
> Luke

it will stop ( well actually sql crash , and it would kill any other program
on the server  that tries to open a cursor ) after 300 cursors are left open.
not the way to run on a production system.

steve
Chris Smith - 05 Feb 2005 03:09 GMT
> it will stop ( well actually sql crash , and it would kill any other program
> on the server  that tries to open a cursor ) after 300 cursors are left open.
> not the way to run on a production system.

Steve, you're a little late here, but just to clarify: that's not true.  
Omitting rs.close() in the code mentioned here will not leave open
cursors anywhere, because the cursor will be closed when the next loop
iteration calls PreparedStatement.execute().  The last cursor is closed
when the connection is closed.

It is possible to avoid this behavior, using the prepareStatement
overload (new in 1.4) that takes an additional parameter on result set
holdability.  Since this is not used in the OP's code, your concern is
not valid.  It turns out that there's very rarely a reason to explicitly
close ResultSet instances, unless you went out of your way to keep them
open in the first place.

In any case, Alex already has a solution to his problem.

Signature

www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

Lee Fesperman - 05 Feb 2005 03:27 GMT
> > it will stop ( well actually sql crash , and it would kill any other program
> > on the server  that tries to open a cursor ) after 300 cursors are left open.
[quoted text clipped - 5 lines]
> iteration calls PreparedStatement.execute().  The last cursor is closed
> when the connection is closed.

That's what the spec says, but I think that closing resultsets is needed for some
backends (Oracle). Also, you should close the prepared statement instead of waiting for
the connection to do the job. IOW, the best practice is always close resultsets,
statements and connections and do it as soon as you don't need them anymore. For a
decent backend, this should not be more costly and will free up resources on the server
... to everyone's benefit.

Signature

Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com)
==============================================================
* The Ultimate DBMS is here!
* FirstSQL/J Object/Relational DBMS  (http://www.firstsql.com)

Chris Smith - 05 Feb 2005 04:43 GMT
> That's what the spec says, but I think that closing resultsets is needed for some
> backends (Oracle).

You're saying that Oracle is providing terribly broken JDBC drivers that
cause a normal and correct JDBC application to cause severe problems on
their server, and hasn't fixed this in nearly 10 years?  That would
really be rather astonishing, quite frankly.  Any reference to provide
evidence for that?

Signature

www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

Joe Weinstein - 05 Feb 2005 04:51 GMT
>>That's what the spec says, but I think that closing resultsets is needed for some
>>backends (Oracle).
[quoted text clipped - 4 lines]
> really be rather astonishing, quite frankly.  Any reference to provide
> evidence for that?

It's a function of DBMS-side cursors which are associated with open statements
and open result sets. It is common, correct JDBC hygiene to close such objects
ASAP, and for Oracle it is more important than for some DBMSes. Oracle permits
a fixed (configurable) number of cursors that a client can have open at one
time. An application that caches or ignores too many open statements or result
sets can get errord if it exceeds the DBMS limit. It's not a JDBC issue as such.

Joe Weinstein at BEA
Chris Smith - 05 Feb 2005 05:31 GMT
> It's a function of DBMS-side cursors which are associated with open statements
> and open result sets. It is common, correct JDBC hygiene to close such objects
> ASAP, and for Oracle it is more important than for some DBMSes. Oracle permits
> a fixed (configurable) number of cursors that a client can have open at one
> time. An application that caches or ignores too many open statements or result
> sets can get errord if it exceeds the DBMS limit. It's not a JDBC issue as such.

Apparently you haven't been reading the thread.  Oracle's JDBC driver
should not be holding a ResultSet open across a Connection.close() --
nor a commit() unless a non-default holdability was set using JDBC3's
new overloads of createStatement and prepareStatement.  Nor should it
survive a Statement.execute() or getMoreResults() (the version with no
parameters).  If the Oracle driver is producing ResultSet instances that
sutvive these things, then it is in violation of the spec, including
(but not limited to) sections 10.1, 14.1.3, 14.2.5, Oracle's driver is
broken, and it's broken in a particularly gruesome way.

The spec (somewhat confusingly, and for no good reason that I can see)
allows for greater leeway for Statement.close() -- requiring the driver
to immediately invalidate and "close" the ResultSet, but permitting it
to hold off on releasing resources until the next garbage collection.  
That's not a particularly useful allowance, and I imagine that there are
probably no drivers that take advantage of it but still comply with the
spec.  It appears that the author of the specification does not actually
understand garbage collection in Java.  That theory is confirmed when
section 14.2.5 specifies that calling ResultSet.close() makes the
ResultSet object immediately available for garbage collection -- a
ludicrous requirement that invalidates the entirety of the Java security
model, and requires a heavily modified virtual machine to actually
implement correctly (luckily, no one actually takes this requirement of
the spec seriously).

Clearly that is in fact a JDBC issue, since it's about whether Oracle's
driver complies certain requirements of the JDBC specification or not.  
If it is confirmed that this is a problem with Oracle's JDBC driver,
then apparently it will become necessary to make calls to
ResultSet.close() in the middle of a tight loop of PreparedStatement's
execute() calls.  That would then be classified as a work-around for a
serious bug, *not* as good general practice for JDBC.

Signature

www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

Joe Weinstein - 05 Feb 2005 05:55 GMT
> Apparently you haven't been reading the thread.

you're right, sorry.

  Oracle's JDBC driver
> should not be holding a ResultSet open across a Connection.close() --
> nor a commit() unless a non-default holdability was set using JDBC3's
> new overloads of createStatement and prepareStatement.

I'll catch up on the thread. Is that happening?

> Nor should it
> survive a Statement.execute() or getMoreResults() (the version with no
> parameters).  If the Oracle driver is producing ResultSet instances that
> survive these things, then it is in violation of the spec, including
> (but not limited to) sections 10.1, 14.1.3, 14.2.5, Oracle's driver is
> broken, and it's broken in a particularly gruesome way.

I agree. When last I checked, the result sets that came from standard
calls, such as executeQuery() *were* closed properly when the statement
was closed or re-executed, but those result sets that arrived via
non-standard JDBC, such as via output parameters of procedures,
via their non-standard CallableStatement.getResultSet( int paramIndex)
were not closed. A bug indeed, if it's still true. In fact the driver
used to allow a user to execute a stored procedure via executeQuery()
which must return a result set. The result set was some odd bogus result
set that didn't relate to the procedure output.

> The spec (somewhat confusingly, and for no good reason that I can see)
> allows for greater leeway for Statement.close() -- requiring the driver
[quoted text clipped - 10 lines]
> implement correctly (luckily, no one actually takes this requirement of
> the spec seriously).

Yep, silly. I still have a reference to it. *Try* gc()'ing it...
Even if it is closed and the reference is gone, until getMoreResults()
is called, one could conceivably call getResultSet() again and expect to
get the same already-closed result set, so the statement has to keep a
reference for a while too.

> Clearly that is in fact a JDBC issue, since it's about whether Oracle's
> driver complies certain requirements of the JDBC specification or not.  
[quoted text clipped - 3 lines]
> execute() calls.  That would then be classified as a work-around for a
> serious bug, *not* as good general practice for JDBC.

I always close everything asap... Chaqu'un a son gout... but yes, I wasn't
reading the thread.
Joe Weinstein at BEA
Chris Smith - 05 Feb 2005 14:09 GMT
>    Oracle's JDBC driver
> > should not be holding a ResultSet open across a Connection.close() --
> > nor a commit() unless a non-default holdability was set using JDBC3's
> > new overloads of createStatement and prepareStatement.
>
> I'll catch up on the thread. Is that happening?

I don't know.  Lee suggested that it is happening.  I hope he's wrong.

> Yep, silly. I still have a reference to it. *Try* gc()'ing it...
> Even if it is closed and the reference is gone, until getMoreResults()
> is called, one could conceivably call getResultSet() again and expect to
> get the same already-closed result set, so the statement has to keep a
> reference for a while too.

I agree with all but the getResultSet() piece.  getResultSet() is only
guaranteed to work once per result.  The very first time you call it for
a given result, the Statement is entirely within its rights to throw
away any references to that ResultSet.  After that, it can throw
SQLException instead of returning a ResultSet.

> I always close everything asap... Chaqu'un a son gout... but yes, I wasn't
> reading the thread.

I certainly would not argue for leaving things open indefinitely.  
Specifically, though, we're discussing whether it's reasonable to omit a
call to ResultSet.close() in the middle of a tight loop of calls to
PreparedStatement.executeQuery().  As far as I'm concerned, that IS
basically closing things ASAP; the next call to executeQuery is going to
happen immediately, and it will close the ResultSet.  Unless Lee is
right about Oracle's driver bug...

Signature

www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

Dimitri Maziuk - 05 Feb 2005 18:51 GMT
Chris Smith sez:
>>    Oracle's JDBC driver
>> > should not be holding a ResultSet open across a Connection.close() --
[quoted text clipped - 4 lines]
>
> I don't know.  Lee suggested that it is happening.  I hope he's wrong.

Yep, he is. I don't know about result sets returned from stored
procedures, but regular ones get closed as per specs (at least
in 8i and 10g).

IME it's statement.close() that releases server-side cursor;
you want to explicitly close your statements ASAP, if you leave
that to gc you're likely to run out of cursors. Moreover, it
seems that gc'ing unclosed statements does not always close
server side of the connection, sometimes it sits there hours
waiting to time out.

>> I always close everything asap... Chaqu'un a son gout... but yes, I wasn't
>> reading the thread.
[quoted text clipped - 5 lines]
> basically closing things ASAP; the next call to executeQuery is going to
> happen immediately, and it will close the ResultSet.

I always close the statements and usually close associated result
set just before stat.close(), just to be sure. I never close the
result set in a middle of a loop (tight or not) and so far I haven't
run into any problems.

... Unless Lee is
> right about Oracle's driver bug...

There's 2 thin drivers in 10g: new ...OracleDriver and old
...driver.OracleDriver. The old one is retained for compatibility
backwards, all bugfixes and new features go into new one. Plus
there's not-thin driver(s?), RowSet implementation jar, etc.
There may be that Lee's bug is in one or two of them, who knows...

Dima
Signature

Riding roughshod over some little used trifle like the English language is not a
big deal to an important technology innovator like Microsoft. They did just that
by naming a major project dot-Net (".Net").  Before that, a period followed by a
capital letter was used to mark a sentence boundary. --T. Gottfried, RISKS 21.91

Lee Fesperman - 05 Feb 2005 21:05 GMT
> > That's what the spec says, but I think that closing resultsets is needed for some
> > backends (Oracle).
[quoted text clipped - 4 lines]
> really be rather astonishing, quite frankly.  Any reference to provide
> evidence for that?

Don't be naive. Do you really think Oracle gives a fig about compatibility with specs?
SQL Server is much worse, and DB2 is not so hot. These are the 800 pound gorillas. If
you want to use them, you have to put up with their eccentricities. After that there is
a fairly large number of other JDBC drivers, including more than one for some DBMSs and
then there are different versions. Add all the ODBC drivers accessible through JDBC-ODBC
bridges, you'll find that compatibility is all over the map.

Remember that the next time you recommend a feature like resultset holdability (new in
JDBC 3). The rate of adoption is slow and spotty. Some will never support new features.

At least with JDBC, much of this can be covered with defensive programming. Have you
ever written a JDBC application intended to run with a variety of drivers and backends?
I have (several in fact). It is possible to get reasonable coverage if you're careful.
The situation with SQL is much worse. Everyone has their own proprietary version of the
language.

Signature

Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com)
==============================================================
* The Ultimate DBMS is here!
* FirstSQL/J Object/Relational DBMS  (http://www.firstsql.com)

Dimitri Maziuk - 06 Feb 2005 18:03 GMT
Lee Fesperman sez:
...
> At least with JDBC, much of this can be covered with defensive programming. Have you
> ever written a JDBC application intended to run with a variety of drivers and backends?
> I have (several in fact). It is possible to get reasonable coverage if you're careful.
> The situation with SQL is much worse. Everyone has their own proprietary version of the
> language.

I love how Oracle's thin driver doesn't have
DatabaseMetadata.getDatabaseXXXVersion(). So that you can't do
if( version >= 10 ) stat.executeUpdate( "drop table foo purge" );
else stat.executeUpdate( "drop table foo" );
without a pattern match on the product name string.
Hatess 'em, we doess...

Dima
Signature

Backwards compatibility is either a pun or an oxymoron.                  -- PGN

Lee Fesperman - 06 Feb 2005 19:39 GMT
> Lee Fesperman sez:
> ...
[quoted text clipped - 10 lines]
> without a pattern match on the product name string.
> Hatess 'em, we doess...

Interesting. What does "drop table foo purge" mean?

Signature

Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com)
==============================================================
* The Ultimate DBMS is here!
* FirstSQL/J Object/Relational DBMS  (http://www.firstsql.com)

Dimitri Maziuk - 07 Feb 2005 16:55 GMT
Lee Fesperman sez:
...
> Interesting. What does "drop table foo purge" mean?

It doesn't delete tables anymore, it renames them to
RECYCLEQWJQ#(*$@!#JR.BIN instead. "Purge" means
"don't clutter my tablespace with junk, I really mean
to delete the fscking thing and I know what I'm doing".

Dima
Signature

Tlaloc: What was Elrond's second name?
Gruber: Hubbard                           -- <ahbou=3C69EB63.A7C431F4@last.com>

Lee Fesperman - 07 Feb 2005 20:24 GMT
> Lee Fesperman sez:
> ...
[quoted text clipped - 4 lines]
> "don't clutter my tablespace with junk, I really mean
> to delete the fscking thing and I know what I'm doing".

Thanks for the explanation. The 800 pound gorilla at work again ;^)

Signature

Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com)
==============================================================
* The Ultimate DBMS is here!
* FirstSQL/J Object/Relational DBMS  (http://www.firstsql.com)

Robert Klemme - 01 Feb 2005 13:54 GMT
> Luke:
> >we're trying to help
[quoted text clipped - 22 lines]
> oracle.jdbc.ttc7.TTC7Protocol.createDBItem
> 9  0.50% 85.42%       6   158 java.lang.StringBuffer.toString

Did you feed that into an analysis tool like HPJmeter?  Maybe there's
something different going on on the network.

Also, for the db it might be more efficient if you include the data you
draw from variable "table" and execute a slightly modified SQL just once.

   robert

> I do not believe in profiling. It takes time.
> And during this time Oracle is working too!
[quoted text clipped - 235 lines]
>
> Alex Kizub.
akizub@yahoo.com - 01 Feb 2005 14:48 GMT
>Did you feed that into an analysis tool like HPJmeter?
I used HPjmeter-1.6. It's a little old. Latest version is for 2003. And
supports
JAVA PROFILE 1.0 when latest java produces JAVA PROFILE 1.0.1.
Nevertheless you can see original text file which is obvious enough.

>Maybe there's something different going on on the network.
Problem is that both perl and java are working in the same environment,
the same time,
the same database and the same sql. And result is always the same.
perl at least twice faster. If java runs 90 minutes then perl runs 25.
If java runs 43 minyes then perl 17. And so on.

>Also, for the db it might be more efficient if you include the data you
draw from variable "table" and execute a slightly modified SQL just
once.

Of course I can modify SQL. But I have to do it for both langauges. So
this is not the point for difference. And, I assume, that 2000 SQLs are
good enough for statistic's measure.

Disappointing point is that if I use simple SQL like "select 1 from
dual' then both languages have the same result. Which means for me that
overhead charges are the same. When I change results (like increase
key numbers for hastable, make them unique = 2000 all different, but
for the same simple query) then java is faster.
Which means calculation itself are better in java.

But database connection kills everything.
I had another experience with IBM original JDBC connection to mainframe
DB2.
select for 1,000,000 items table was about 5 minutes.
But I thought that IBM forced programers to use COBOL not Java.
And gave up for it too. My Java world shrinks so rapidly....
:(((((((((((((((

Alex Kizub.
dar7yl - 01 Feb 2005 22:48 GMT
Alex:

I think I can see where a lot of your time is being spent - in the sql query
and the subsequent returning of the resultset.

If, as you say, the query is generated the same in java as it is in perl,
then the only differences would be the transmission of the resultset,
as the sql server would be performing the same operation
(nearly enough) in both cases.

If your jdbc driver supports it, turn on tracing/profiling.   The MySQL
driver has this feature which you turn on by adding "?&profileSql=true"
to the connect string.  This is a great feature, which shows the actual
expanded query, the server execution time, and the transmission time.

I assume that the query returns only one result.  It would be better
programming practice to surround the logic with
   if (rs.next())
   {
       // logic goes here
   }
instead of performing the continue on the negative case.  (bad style).
Even better would be to make that a while, just in case the query
returns more than one result.

Perhaps, you could optimize the query to return the whole list in one call.
A way to do that would be to create a temporary table from your getData()
table and then do a join in your sql query, thus returning a dataset with
more than
one row.  This would take a great deal of load off the server, as it would
only
have to perform the scan once (albeit a more complex query).

regards,
   Dar7yl

> Luke:
>>we're trying to help
[quoted text clipped - 262 lines]
>
> Alex Kizub.
akizub@yahoo.com - 02 Feb 2005 02:10 GMT
Dar7yl:
Thank for answer.
>If your jdbc driver supports it, turn on tracing/profiling.
I use jodbc14.jar from Oracle. Don't know how to force it to trace.
Tomorrow will have a look on oracle.com.
But what is the diffence where time is spent? I still have to pay the
price.

>     if (rs.next())
>    {
>        // logic goes here
>    }
Of course I usually do this. You can see it from previous code.
I only tried to simplify the code and reduce number of getMilliseconds
calls.

>Perhaps, you could optimize the query to return the whole list in one call.

Of course I do it in real project.
And I can do it in both Java or perl.
But then quiry eats so much power of Oracle that DBAs call me and ask
to stop it.
Did you ever had such experience when your query runs for hour and DBA
call you and ask what is that query which combines 9 tables and hold
whole system for an hour?
What can I answer for such DBA? That this is your database structure
and your relations? And there is no other way to obtain data from your
imparctical design?
And people like this design and support this system...
So, I ask no questions.
I ask them here :))))

Anyway. It seems for me more and more that difference is in OCI driver
for perl. I'll try to play with it but will take time.
And anyway again it's rear approach. Most applications use thin driver.
And when it three times slower then OCI... too bad.

Thanks anyway. I already gave up.
But it was good experience for me.
I know a lot of new stuff and will know even more (with this OCI I
hope).

And project remains on perl. But who cares!
perl is much better then COBOL.
Another of my previous projects is still on COBOL :(((

Alex Kizub.
Tom Dyess - 02 Feb 2005 03:03 GMT
> Dar7yl:
> Thank for answer.
[quoted text clipped - 44 lines]
>
> Alex Kizub.

Can't you exp the schema/tables and imp it onto another server and tune it
without bothering anyone else? Don't you have a development box at work?
There is something really wrong going on and you're gonna have to tinker
with it. One of us could probably find it after a few minutes if we could
connect to your environment, but in newsgroups, its just conjecture and
theory. Try putting a /*+ RULE */ hint in your SQL (after the first select)
and make sure your indexes are in place. You're connected to the proper
instance - i.e. there aren't two instances on the same box and you're
connecting to one via perl and the other via Java are you?

Signature

Tom Dyess
OraclePower.com

akizub@yahoo.com - 02 Feb 2005 13:12 GMT
>Can't you exp the schema/tables and imp it onto another server and tune it
without bothering anyone else?
No. On devl it works fine.

> Don't you have a development box at work?
Our DBAs can't replicate situation in devl. It exists only in prod.

>There is something really wrong going on
I totally agree.

> and you're gonna have to tinker with it.
And that's what I'm doing.

> One of us could probably find it after a few minutes if we could
connect to your environment,
It's not even mine.

>its just conjecture and  theory.
And that's what I'm talking about.

> Try putting a /*+ RULE */ hint in your SQL (after the first select)
and make sure your indexes are in place.
This is not about tune or performance. This is about comparisond two
languages.
So, everything should be the same. Bad or good.

>You're connected to the proper
instance - i.e. there aren't two instances on the same box and you're
connecting to one via perl and the other via Java are you?

Are you kidding? Everything is the same.

Sorry. Couldn't resist to answer to guy from  OraclePower.com
Even topic is closed.
Sincerely,
Alex Kizub.
Tom Dyess - 02 Feb 2005 13:27 GMT
> >Can't you exp the schema/tables and imp it onto another server and
> tune it
[quoted text clipped - 33 lines]
> Sincerely,
> Alex Kizub.

Yea, I just don't like seeing it die. I like to solve problems but looks
like this one is taking a dirt nap. I thought the objective was to make the
Java version run at a comperable speed to Perl as opposed to more of an
academic nature. To do that, it requires more than your environment is
willing to let you do. Sounds like your DBA won't even let you work on it
much less give you explain plans, so you're kinda feeling in the dark. Oh
well, it was fun anyway.

Tom Dyess
OraclePower.com
Alex - 02 Feb 2005 21:10 GMT
My apology. I became victim of circumstances and my own ambitions.
That's production database and performance varies too much.
Right now I did a lot of other statistics and discovered that in
average SQL time varies 100 times.
So, when I saw that perl was faster (and it was a little faster because
used OCI driver) and after 3 to 5 runs (remember, they take about hour)
I changed SQL from plain one to prepared statement. So, when perl sent
new SQL every time Java used parameters in predefined one. And it
became even slower.
I was upset and started this topic.

Now DBA (other, not ours) did a little research and discovered that
Explanation Plan which Oracle uses for prepared statement is totally
different then for plain text SQL. And based on whatever it was based
this plan in reality was much worse that plan for the same SQL but with
values except parameters.

So, I changed SQL back from prepared to generated for each run (just as
perl did) and got approximately the same results for both languages.
Both, again, were different time to time at least 10 times but, at
least similar. Sometime j