Java Forum / Databases / February 2005
Is perl better? :(((
akizub@yahoo.com - 30 Jan 2005 15:34 GMT I'm proud and certified Java programmer since 1998. Trust me, a lot of experience! And I prefer Java. Also I work with other languages too. For example we support/develop big system with around 300 perl scripts. I try to move them to Java, but...
Recently I tried port one perl script. It does nothing but creates thousands SQLs retrieve and store/summarize rezults. SQLs are complicated a little but return only pair of values. And this pair is stored in hashtable. That's all. Simple enough.
Why not to rewrite? But in Java it runs at least twice slower! In my understanding there is no Java/perl calculations. Store couple values! Phoo!
All time, I assume, is wasted on Oracle 9 connection. In middleware. The SQLs are the same, connection is the same, Oracle is the same, machine is the same. Everything is the same. Except time. perl is faster somehow.
Then I used prepared statements. It became worse. At least 2.5 times slower... What's the deal? Any ideas why it happens?
I believe I use latest Java 1.4, JDBC (ojdbc14.jar) from Oracle as well as latest perl DBD or Oraperl modules. Who have better experience?
Alex Kizub.
KiLVaiDeN - 30 Jan 2005 17:42 GMT > I'm proud and certified Java programmer since 1998. > Trust me, a lot of experience! And I prefer Java. [quoted text clipped - 28 lines] > > Alex Kizub. Do you use Connection pool ? maybe the time involved is lost in connecting all the time
K
akizub@yahoo.com - 30 Jan 2005 18:11 GMT One connection. Always alive. All these hours :((( Alex Kizub
Robert Klemme - 30 Jan 2005 18:36 GMT > One connection. Always alive. All these hours :((( > Alex Kizub If the SQL is exactly the same and you used the same machine for boths tests the most likely cause is probably your Java code. Did you profile it?
robert
akizub@yahoo.com - 30 Jan 2005 18:54 GMT 100%
Chris Smith - 30 Jan 2005 21:38 GMT > 100% Huh? Do you mean it's using 100% of the CPU? If so, then you *definitely* need to aquire a profiler and figure out where that time is being spent. It shouldn't be taking nearly so long to issue queries. There are a number of mistakes you could be making, but without seeing your code or your profiling results, there's little that we can do.
 Signature www.designacourse.com The Easiest Way To Train Anyone... Anywhere.
Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
akizub@yahoo.com - 30 Jan 2005 22:50 GMT Sorry. 100% that all is the same. 100% that Java is not wasting the time. And I'm sure in 100% that it is something with JDBC and Java middleware.
See my code in this tread. I posted it twice altready. And it's so pity when I open two windows for the same Solaris and see that every 100'th SQL comes from perl in 5 seconds when Java produce the same answer in the same exactly time and for the same exactly SQLs (so there is no Oracle cache or hints or statistics, tune and so on are different) 2-3 times slower. And, trust me. There is no more pain for me that type the same perl command in perl windows 3 times when it is still first run for Java and for exactly the same result total.
Loosing hope. Alex Kizub.
Chris Smith - 31 Jan 2005 17:43 GMT > 100% that all is the same. > 100% that Java is not wasting the time. > And I'm sure in 100% that it is something with JDBC and Java > middleware. So, in other words, you haven't bothered to use a profiler yet. Stop wasting eveyone's time, then. If you wish to struggle against the unknown and refuse to use tools that will help you, that's your problem. Just go away and do it on your own. It's becoming clear that you aren't willing to work with anyone to solve your problem.
 Signature www.designacourse.com The Easiest Way To Train Anyone... Anywhere.
Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
Juha Laiho - 30 Jan 2005 19:25 GMT akizub@yahoo.com said:
>Recently I tried port one perl script. It does nothing but creates >thousands SQLs retrieve and store/summarize rezults. SQLs are [quoted text clipped - 3 lines] >Why not to rewrite? >But in Java it runs at least twice slower! If I recall correctly, JDBC will at some point do conversion from the database character set to Unicode - and perl will not, unless your environment is set up to use/require use of UTF8 character encoding. This could be at least part of what you're seeing.
Are your programs CPU-bound? That is, are they using (almost) one second of CPU time for each wallclock second? If so, that'd sound even more like the problem would be in character set conversions.
Btw, is it the JDBC thin driver you're using? You might wish to compare results when using JDBC OCI driver; there could be a difference. The perl DBD::Oracle driver uses the OCI driver layer anyway.
 Signature Wolf a.k.a. Juha Laiho Espoo, Finland (GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++ "...cancel my subscription to the resurrection!" (Jim Morrison)
Dimitri Maziuk - 30 Jan 2005 19:45 GMT akizub@yahoo.com sez: ...
> All time, I assume, is wasted on Oracle 9 connection. In middleware. > The SQLs are the same, connection is the same, Oracle is the same, [quoted text clipped - 5 lines] > What's the deal? > Any ideas why it happens? Oracle's JDBC driver is very fast -- I have a Java program that uses prepared statements to bulk-load data from text file, it's as fast as using oracle loader.
You can reasonably expect Java to be a little slower, but not much. A 2.5 times slowdown with prepared statements is a sure sign that your code is broken.
Dima
 Signature Things seemed simpler before we kept computers. -- IX, Revelation
steve - 30 Jan 2005 21:31 GMT > akizub@yahoo.com sez: > ... [quoted text clipped - 17 lines] > > Dima oracles jdbc is not fast, it is a piece of crap. try using 1. the demo version from datadirect ,it is good for 15 days, but will allow you to bench mark 2. try using the thin driver from java. 3. post your code ( minus passwords & domain), ill take a look.
steve
akizub@yahoo.com - 30 Jan 2005 22:36 GMT >oracles jdbc is not fast, it is a piece of crap. Looks like this for me now :(((
>1. the demo version from datadirect ,it is good for 15 days, but will allow you to bench mark
See this is production and, actually, third party application. Do you think they will install something new for Java when perl is already faster?
>2. try using the thin driver from java. Don't understand this. I tried few different JDBC thin drivers including latest Oracle ojdbc14.jar for our particlular Oracle9i 9.2.0.4 JDBC Drivers.
>3. post your code ( minus passwords & domain), ill take a look. See another anwers in this thraed. Pretty simple. Here is code. Exactly. Except the SQL. Hashtable ht=new Hashtable(); PreparedStatement ps = conection.getPreparedStatement("select 'key', sysdate+? from dual");
for (int i=0; i<2000; i++) ps.setInt(1, i);
rs = ps.executeQuery(); if (rs.next()) { String key=rs.getString(1); double value=0; Double d=(Double)ht.get(key); if (d!=null) value=d.doubleValue(); value+=rs.getDouble(2); ht.put(key,new Double(value));
} rs.close(); }
Still have a little hope. But less and less... Alex Kizub.
Robert Klemme - 31 Jan 2005 10:22 GMT > >oracles jdbc is not fast, it is a piece of crap. > Looks like this for me now :((( [quoted text clipped - 38 lines] > Still have a little hope. But less and less... > Alex Kizub. It's quite inefficient to recreate Doubles all the time although this is the straightforward approach. You can easily avoid that by creating a mutable double class and put that into the map. This will easily be faster.
Kind regards
robert
akizub@yahoo.com - 31 Jan 2005 13:14 GMT >It's quite inefficient to recreate Doubles all the time It's not big deal. I have 2000 SQL so I have 2000 new Doubles. Sure it's not the point to lost 10 minutes. I'm talking about significant difference.
John C. Bollinger - 31 Jan 2005 15:52 GMT >>It's quite inefficient to recreate Doubles all the time > > It's not big deal. I have 2000 SQL so I have 2000 new Doubles. > Sure it's not the point to lost 10 minutes. > I'm talking about significant difference. You don't know how much of a difference it makes because you refuse to profile. I agree that creating 2000 Doubles ought not to take any significant fraction of 10 minutes, but then your whole test program ought not to take more than a few seconds unless there is a bottleneck elsewhere (e.g. network, DB).
John Bollinger jobollin@indiana.edu
dar7yl - 01 Feb 2005 22:08 GMT Looking at your code, I can see immediately where your problem is. Remember, perl syntax and java syntax are different.
> for (int i=0; i<2000; i++) > ps.setInt(1, i); this loops around 2000 times, setting the same object. time waster supreme. Perhaps you meant to put braces {} around a larger section.
> conection.getPreparedStatement("select 'key', sysdate+? from dual"); I'm not sure that this is the correct form for what you are trying to do. You cannot pass the name of a column in a select clause as a parameter to a prepared statement.
Perhaps you meant to say "select 'key', 'sysdate' from dual." then execute your query, returning a resultset rs, which you iterate so:
ResultSet rs = ps.executeQuery(); while ( rs.next() ) { String key=rs.getString(1 ); double value=0; Double d=(Double)ht.get(key); if (d!=null) value=d.doubleValue(); value+=rs.getDouble(2); ht.put(key,new Double(value)); }
regards, Dar7yl.
> >oracles jdbc is not fast, it is a piece of crap. > Looks like this for me now :((( [quoted text clipped - 38 lines] > Still have a little hope. But less and less... > Alex Kizub. steve - 04 Feb 2005 21:16 GMT >> oracles jdbc is not fast, it is a piece of crap. > Looks like this for me now :((( [quoted text clipped - 6 lines] > Do you think they will install something new for Java when perl is > already faster? no but it wil lallow you to find out the fastest you are EVER going to run.
>> 2. try using the thin driver from java. > Don't understand this. I tried few different JDBC thin drivers > including latest Oracle ojdbc14.jar for our particlular Oracle9i > 9.2.0.4 JDBC Drivers. sorry!! the jdbc has 2 modes. 1. pure java. 2. java callouts to c++ oci system. it depends on how you call the java driver when you first make hte connection.
>> 3. post your code ( minus passwords & domain), ill take a look. > See another anwers in this thraed. Pretty simple. [quoted text clipped - 22 lines] > Still have a little hope. But less and less... > Alex Kizub. see my post else where for problems on your code.
steve
akizub@yahoo.com - 30 Jan 2005 21:40 GMT Here is code. Exactly. Except the SQL. Hashtable ht=new Hashtable(); PreparedStatement ps = conection.getPreparedStatement("select 'key', sysdate+? from dual");
for (int i=0; i<2000; i++) ps.setInt(1, i);
rs = ps.executeQuery(); if (rs.next()) { String key=rs.getString(1); double value=0; Double d=(Double)ht.get(key); if (d!=null) value=d.doubleValue(); value+=rs.getDouble(2); ht.put(key,new Double(value)); } rs.close(); }
That's all. Exceptions are outside and handled once. No existing null values. Hashtable is 1000 or less keys. Not big deal. As well as possible impovements of code like mutable Objects instead Double or similar.
And that's all! No Unicode expenses or huge network trafic. Thin driver. I'll try OCI driver. Always was sure that thin is good enough. Keep guessing! Because it's the shock for me.
Alex Kizub.
John C. Bollinger - 31 Jan 2005 15:56 GMT > Here is code. Exactly. Except the SQL. Evidently it isn't. I observe, for instance, that you misspelled "prepareStatement" as "getPreparedStatement":
> PreparedStatement ps = > conection.getPreparedStatement("select 'key', sysdate+? from > dual"); There is at least one other likely error in your code that has already pointed out elsewhere (the extent of your loop), but I don't know whether that's in the real code or whether it's a typo.
John Bollinger jobollin@indiana.edu
steve - 04 Feb 2005 21:13 GMT > Here is code. Exactly. Except the SQL. > Hashtable ht=new Hashtable(); [quoted text clipped - 28 lines] > > Alex Kizub. loose the first "if" & replace with a while. loose the checks for types inside you loop, replace it with a getobject.
that should get you as fast as you are going to be using the thin driver., using the above code.
but you need to rewrite the whole thing, why loop round a prepared statement 2000 times, that is just madness.
try the following system :
write the code as a serverside pl/sql routine that returns a cursor of 2,000 cols, then call the routine. it will reduce your network calls by atleast 50%, and reduce the servers sh.t down even further.
but i cannot see why you have to do what you are doing , the way you do it.
steve
Chris Smith - 30 Jan 2005 21:42 GMT > You can reasonably expect Java to be a little slower, but not > much. I don't know about that; a little slower than C or FORTRAN in the average case, sure. A little slower than Perl, though? Not unless you tune the test specifically to exploit the strengths of Perl and weaknesses of Java; and even then I wouldn't guarantee you could produce these results easily.
In any case, something is clearly wrong here besides the language.
 Signature www.designacourse.com The Easiest Way To Train Anyone... Anywhere.
Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
Dimitri Maziuk - 31 Jan 2005 17:11 GMT Chris Smith sez:
>> You can reasonably expect Java to be a little slower, but not >> much. > > I don't know about that; a little slower than C or FORTRAN in the > average case, sure. A little slower than Perl, though? JVM startup time and garbage collection.
> In any case, something is clearly wrong here besides the language. Yep. I'd look into CLOSE_CURSORS_ON_COMMIT (or whatever it's called) if I were OP.
Dima
 Signature ... If you want to make sure you don't put a Pig in a List of airplanes and have it fail at insertion rather than extraction, use planelist.add((Airplane)o) instead of planelist.add(o). It's that easy. -- Mark 'Kamikaze' Hughes
Tor Iver Wilhelmsen - 31 Jan 2005 19:15 GMT > JVM startup time and garbage collection. Perl has (in theory at least) both of those as well, though.
Luke Webber - 31 Jan 2005 21:21 GMT >>JVM startup time and garbage collection. > > Perl has (in theory at least) both of those as well, though. And we're talking about a process that runs for as much as ten minutes, so it should not be significant.
Luke
Walter Mitty - 07 Feb 2005 12:54 GMT >>> JVM startup time and garbage collection. >> >> Perl has (in theory at least) both of those as well, though. > > And we're talking about a process that runs for as much as ten minutes, > so it should not be significant. Since he's creating/deleting all the time (probably), it might well be.
Tom Dyess - 30 Jan 2005 21:20 GMT > I'm proud and certified Java programmer since 1998. > Trust me, a lot of experience! And I prefer Java. [quoted text clipped - 28 lines] > > Alex Kizub. Yes, I would have to take a look at the code of the Java application. Was the original PERL script hitting an Oracle database? Why are you calling thousands of SQL statements that just return a pairs? Is there any way to call several of these statements in a single call? PERL is great for text manipulation, but I don't think this fact would cover the gammut of ~2.5x difference in speed. Did you run an explain plan on your Oracle SQL? Is your Oracle box the same DB that the perl scripts are calling? (Ie perl hitting production, java hitting some crappy 90mhz oracle box someone threw together). Are you using SQL hints when tuning your SQL? When you step through a debug, are the SQL calls taking the most time? Are you using the same Oracle client software on both test scenarios? Are you using a persistant connection? (ie not dropping and reconnecting every script). Are you creating a Java application for every script, or are you creating a large single Java app that calls all scripts?
I don't want the answers to all these, just suggesting some things to try. I know you said you were using the same Oracle, but if all you are doing is sending a SQL statement to Oracle, Java is done until you get the result. It doesn't sound the same. By same I mean you are connecting to the same server, same instance with the same client on the same workstation, just two different languages. Does your workstation have enough free memory to be running Java?
On OraclePower.com (ya ya, shameless plug, but I *do* have a point) I run 17 queries on the home page, and it comes back instantly. Oracle hauls-a.s with Java.
 Signature Tom Dyess OraclePower.com
akizub@yahoo.com - 30 Jan 2005 22:28 GMT Tom: Thanks for sharing you experience. I really appreciate it. Especially these are question which I would ask too. As you already read I have big experience too.
> Wasthe original PERL script hitting an Oracle database? Exactly the same.
> Why are you calling thousands of SQL statements that just return a pairs? I hate such design too. Let say in another query from different database (and different time) we have these pairs.
>Is there any way to call several of these statements in a single call? No. pairs are different.
> PERL is great for text manipulation, but I don't think this fact would cover the gammut of ~2.5x difference in speed.
That was my impression too. Just before yetaerday's night.
> Did you run an explain plan on your Oracle SQL? I know I can improve this query. But that's not the point. It will be improved in both languages and it still will be no difference. SQLs are exactly the same. Stupid, impractical but the same.
> Is your Oracle box the same DB that the perl scripts are calling? (Ie perl hitting production, java hitting some crappy 90mhz oracle box someone threw together).
Oracle on one box, Java and perl on another. Both production.
> Are you using SQL hints when tuning your SQL? Yes and no. The same result. See previous answer about SQL.
> When you step through a debug, are the SQL calls taking the most time? Don't use debug only start and end time. Code is simple. See my another answer in the thread.
>Are you using the same Oracle client software on both test scenarios? Yes.
> Are you using a persistant connection? (ie not dropping and reconnecting every script). Yes. One connection and it's alive for hours. No bad connections or another exceptions. Results are the same in perl and Java. But perl runs 2.5 faster. I run two windows on the same Solaris box. In one I run java. In another in the same time (it's about one hour) I can run perl program twice for the same result and have another 10-15 minutes for third run.
> Are you creating a Java application for every script, or are you creating a large single Java app that calls all scripts? All SQLs are in one thread and in one cycle. One by one. Just like perl. Of course creating of thousand of threads could be more useful. But what about poor Oracle? To be honest, our DBAs already called me and asked to stop my SQL and run in the middle of the night. It was first call from them in 5 years.
>I don't want the answers to all these, just suggesting some things to try. I tried these and a lot of others. BECAUSE I DON"t WANT TO USE PERL! I like it too, but support of hundreds of scripts is a nightmare! And what arguments do I have to switch to Java? 20 minutes vs. hour!
>I know you said you were using the same Oracle, but if all you are doing is sending a SQL statement to Oracle, Java is done until you get the result. It doesn't sound the same. By same I mean you are connecting to the same server, same instance with the same client on the same workstation, just two different languages.
Don't use threads. As well as perl. See previous answer.
>Does your workstation have enough free memory to be running Java? This is Solaris with 8Gb. Sure enough to store 1000 keys Hashtable.
>On OraclePower.com (ya ya, shameless plug, but I *do* have a point) I run 17 queries on the home page, and it comes back instantly. Oracle hauls-a.s with Java.
Sorry, don't understand. A lot of resources. Don't know how they can help me. Mostly I know all this and always was sure that Oracale loves Java as well as me. But it doesn't help me to prove my point for managers. Actually what point? Fact is fact. I can't deny it even I don't like it.
Maybe it's OCI. But I can't install it on produstion server. I should use what already exists. So, perl has OCI, JDBC doesn't and perl forever? Very sad Certified and with huge experience Java programmer Alex Kizub.
Tom Dyess - 30 Jan 2005 22:58 GMT > Tom: > Thanks for sharing you experience. I really appreciate it. [quoted text clipped - 98 lines] > Very sad Certified and with huge experience Java programmer > Alex Kizub. The point with the Oraclepower comment was that it is written in Java and runs very fast with Oracle. It was also a shameless plug, so you can ignore it. Lol. I'm using oraclejdbc14.jar and connect using this string:
jdbc:oracle:thin:" + username + "/" + password + "@" + server + ":" + port + ":" + sid;
using this driver:
oracle.jdbc.driver.OracleDriver
I would suggest stepping through your code and see if there is any distinguishable pause between the line before the execution and after. This may help to determine if its the connection or the post-processing that is making the difference. I would guess the latter, but it's better to not guess at this point.
I'm gonna see if I can optimize the code a little, but I'm relatively new to Java compared to Delphi, so I don't know that I will be able to do much.
Tom Dyess OraclePower.com
Tom Dyess - 30 Jan 2005 23:19 GMT >> Tom: >> Thanks for sharing you experience. I really appreciate it. [quoted text clipped - 122 lines] > Tom Dyess > OraclePower.com Ok, after looking at the code, I see some mistakes, so would you send the exact code including the SQL statement? For example
for (int i=0; i<2000; i++) ps.setInt(1, i);
rs = ps.executeQuery();
I think you mean to put the rs = ps.executeQuery() in a block when it currently isn't:
for (int i=0; i<2000; i++) { ps.setInt(1, i);
rs = ps.executeQuery(); }
Please send the exact code so we can run it and establish a time baseline before tuning. Also,
connection.getPreparedStatement("select 'key', sysdate+? from dual");
If you are using java.sql.Connection, it's prepareStatement(String), but I don't know if you have the java.sql.Connection wrapped in an aggregating object or not (I do).
 Signature Tom Dyess OraclePower.com
kjc - 31 Jan 2005 06:08 GMT >>>Tom: >>>Thanks for sharing you experience. I really appreciate it. [quoted text clipped - 163 lines] > don't know if you have the java.sql.Connection wrapped in an aggregating > object or not (I do). Dude, I think you badly want to retract your comments. The code posted by the OP is correct along one dimension. He is setting a query value 2K times FIRST, THEN, issuing the query.
Your code example sends a query 2K times to the DB server. Defintely, NOT a good idea.
Tom Dyess - 31 Jan 2005 13:58 GMT >>>>Tom: >>>>Thanks for sharing you experience. I really appreciate it. [quoted text clipped - 170 lines] > Your code example sends a query 2K times to the DB server. > Defintely, NOT a good idea. Dude! Thanks for that memo from the department of the obvious. Lol. I'm trying to determine which he wants. Trying to understand the idea behind what he is trying to do without actually seeing the full code. Based on his comments, hitting the SQL 2000 times seems like that's what he wants to do, which is yes, a bad idea, but if that's a requirement, that's a requirement. In that case, NO I don't want to retract the statement, because to troubleshoot, you need to completely understand a) what's currently happening, b) what the desired effect is and in this case c) how to achieve the desired effect in a much faster manner.
Consider this snippit: Tell me, what's the point of setting a prepared statement 2000 times without executing the statement? Does that seem strange to you? It does to me. When I see that and not the proper factory creation for a prepared statement, connection.prepareStatement(String), it sends a red flag to which I then presented that reply for further clarification. I wasn't suggesting running the query 2000 times to increase the speed, I was merely asking if that is what he wanted based on the code that I saw. KJC, please read the entire thread first before blurting out. Alex, please clarify.
conection.getPreparedStatement("select 'key', sysdate+? from dual");
for (int i=0; i<2000; i++) ps.setInt(1, i);
rs = ps.executeQuery();
Tom Dyess OraclePower.com
akizub@yahoo.com - 31 Jan 2005 14:30 GMT Sorry. Of course I mistyped this code and forgot one { after for. I think it's obvious because I had odd number of braces and there is no sense to set parameter 2000 times and then do one execution.
OK. Here is "exact" (almost) code. Point is not in few new Double or SQL tune. It is exactly the same in perl. Point is significant difference in time. Java is about 2.5 slower! Probably this is OCI Orcale connection for perl. I can't install it for JDBC (don't have rights).
If I use more simple SQL (something like select sysdate from dual) then perl and Java have the same time. Which is obvious because bottle neck is Oracle. But even in this example Oracle is main time eater! And why Java realization is slower I have no idea.
//================ DA.java =============== import java.sql.*; import java.util.*; import java.io.*; import java.net.*;
public class DA { String url; String user; String password; public Connection con; public Statement stmt;
/** Load Oracle driver one time for application */ static { try { Class.forName("oracle.jdbc.driver.OracleDriver"); } catch (Exception ex) { System.out.println("static driver not found: " + ex.getMessage()); } }
public DA () { }
public String openCon(String url, String user, String password) { if (url instanceof String && url.length() > 0 && user instanceof String && user.length() > 0 && password instanceof String && password.length() > 0) { this.url = url; this.user = user; this.password = password; try { con = DriverManager.getConnection(url, user, password); if (con == null) return "ERROR: DBAccess: connection is null!"; stmt = con.createStatement(); if (stmt == null) return "ERROR: DBAccess: statement is null!"; } catch (Exception ex) { System.out.println( "Connection failed " + ex.getMessage() + "\n" + url + "\n" + user + " " + password); return "Connection failed " + ex.getMessage() + "\n" + url + "\n"; } return "OK"; } else return "Error."; }
/** Close current database connectionand statements */ public String closeCon() { try { stmt.close(); con.close(); return "OK"; } catch (Exception ex) { return ("close connection " + ex.getMessage()); }
}
public static void main(String[] args) { System.out.println("DBAccess test START!");
new DA1().run();
System.out.println("DBAccess test END! Java="+System.getProperty("java.version","Not defined"));
}
public void run() { openCon("jdbc:oracle:thin:@yahoo.com:1521:users", "scott","tiger");
try { ResultSet rs = null; PreparedStatement ps = getPreparedStatement("select sysdate+? from dual"); ps.setInt(1, 365);
rs = ps.executeQuery(); if (rs.next()) { System.out.println("Prepared statement " + rs.getString(1)); } rs.close();
} catch (Exception ex) { System.out.println("SQL run Exception " + ex.getMessage()); }
closeCon(); }
public PreparedStatement getPreparedStatement(String sql) throws java.sql.SQLException { return con.prepareStatement(sql); }
//////////// method itself /////////////// public Vector table; // vector of String[5]; 2000 elements // collecting code, month, userid, start, end, public Hashtable totaled_hrs=new Hashtable(); public void getData() { try{ DA da=new DA(); ResultSet rs; Properties pr=new Properties(); pr.load("/config/common.properties");
da.url=pr.getProperty("URL"); da.user=pr.getProperty("User"); da.password=pr.getProperty("Password");
if (!"OK".equals(da.openCon(da.url,da.user,da.password))) System.out.println( "Error1");
double value=0; String tmp[]=null; double total=0,time=0; String month="00";
String sql="SELECT SUM(DECODE(TO_CHAR(e.date, 'MM'), ?, e.hours, 0)) hours, \n"+ "d.person_type \n" + "FROM projects a, tasks b, assignments c, resources d, hours e, \n"+ "super_projects h, users g \n"+ "WHERE \n"+ "d.id = ? \n"+ "AND (e.date >= TO_DATE(?, 'mm/dd/yyyy')) \n"+ "AND (e.date <= TO_DATE(?, 'mm/dd/yyyy')) \n"+ "AND a.id = b.projectid \n"+ "AND b.id = c.taskid \n"+ "AND c.resourceid = d.id \n"+ "AND c.id = e.assignmentid \n"+ "AND e.approved = 'yes' \n"+ "AND a.superid = h.id \n"+ "AND h.name like 'Billed %' \n"+ "AND a.name not like 'Local %' \n"+ "GROUP BY d.person_type \n";
PreparedStatement ps = da.getPreparedStatement(sql);
for(int i=0; i<table.size(); i++) { tmp=(String[])(table.get(i));
ps.setString(1, tmp[1]); ps.setString(2, tmp[2]); ps.setString(3, tmp[3]); ps.setString(4, tmp[4]); rs = ps.executeQuery();
if (rs.next()) { value=rs.getDouble(1);
if ("volunteer".equals(rs.getString(2))) { time=0; } else { time=value; } } total+=time; Double d=(Double)totaled_hrs.get(tmp[0]); if (d!=null) { time += d.doubleValue(); } totaled_hrs.put(tmp[0], new Double(adjusted_summed_time));
} rs.close(); } da.closeCon(); System.out.println("Hours: "+total); } catch (Exception e){ System.out.println( "Error "+e); }
}
} ################# perl ######################## #!/perl/5.6/bin/perl use lib "/perl/5.6/DBD/oracle8.1.6/lib/site_perl/5.6.0/sun4-solaris"; use Oraperl;
... open connection ....
sub getData{
my(@persons)=@_; $total=0; %totaled_hrs=();
foreach $key(@persons){
@one_person=split(/:/,$key);
$sql=...;
$db_list=&ora_open($lda,$sql) || die "\nCan't open cursor 1: $ora_errstr\n";
($time, $type)=&ora_fetch($db_list);
if($type eq "volunteer"){ $time=0; } $totaled_hrs{$one_person[0]}+=$time; $total+=$time;
}
Alex Kizub.
Robert Klemme - 31 Jan 2005 16:14 GMT > Sorry. Of course I mistyped this code and forgot one { after for. > I think it's obvious because I had odd number of braces and there is no > sense to set parameter 2000 times and then do one execution. > > OK. Here is "exact" (almost) code. Apparently not. I suggest you post code that compiles and is properly indented. If you want feedback from people around you have to do at least a bit to make people's life's simpler.
Regards
robert
akizub@yahoo.com - 31 Jan 2005 16:45 GMT >Apparently not. I suggest you post code that compiles and is properly indented.
This code works. Unfortunately, when it was posted to the news/forum it was broken. But you can see the idea. And see critical parts. And see that everything is ok.
When I post short cut from code people want to see whole. When I post whole code people don't like because it's too big. Why do you suspect me that I miss double quote or curly braces and start this big discussion?
Code is simple in both languages. And this is not about - tune SQL - make connection - improve performance - change hardware, software, database or it's structure or algorithm - do my work
Question is simple: Why for the same SQL, hardware, database, network, time and so on for small amount of 2000 SQL perl does it job for 17 minutes when java does for 43?
If you divide these 43 miniutes for 2000 you can see that bottle neck is definitely not couple new Doubles but connection to Oracle. And as you can see SQL are too slow. But why they are faster for perl? There is only one connection, as you can see in source. What slows SQLs so much?
Agian. I don't like this database. It s... I don't like this SQL. I don't like this project which was created ASAP (As Stupid As Possible) and so on so on.
But qestion remains: why it is so significant difference for two languages for all the same environment/etc...?
Alex Kizub.
kjc - 31 Jan 2005 16:56 GMT >>Apparently not. I suggest you post code that compiles and is properly > [quoted text clipped - 37 lines] > > Alex Kizub. Alex, the best thing to do is run it through a profiler. After revewing the output from a profiler, you should have your answer.
Luke Webber - 31 Jan 2005 21:43 GMT >>Apparently not. I suggest you post code that compiles and is properly > [quoted text clipped - 4 lines] > But you can see the idea. And see critical parts. And see that > everything is ok. Nope, not a bit. I've gotta tell tou, that code isn't even close to running. What are all these references to DA1? Where is "table" initialised (hint: it's not). Also, you have too many right-braces in your getData method before the catch statement, "adjusted_summed_time" is not defined, the call to pr.load(InputStream) is passing a filename instead of an InputStream...
> When I post short cut from code people want to see whole. When I post > whole code people don't like because it's too big. > Why do you suspect me that I miss double quote or curly braces and > start this big discussion? Calm down. It's not suspicion, we're trying to help, but if you want us to help, you need to be clear on what you're trying to do. And you've been very unclear to date.
> Code is simple in both languages. Actually, the PERL code is a lot simpler. I don't speak PERL myself, but if that's all there is to the PERL, you're doing a lot more in the Java version.
> And this is not about > - tune SQL [quoted text clipped - 7 lines] > for small amount of 2000 SQL perl does it job for 17 minutes when java > does for 43? Show us working code and maybe we can tell you. Perhaps you could put it up on a web server somewhere for us to download. And /please/ make sure that it's complete, or you're wasting the time of people who are trying to help you. For free, I might add.
Luke
akizub@yahoo.com - 01 Feb 2005 04:31 GMT Luke:
>we're trying to help And I really appreciate it. I'm only a little bit upset when people say that I'm unwiling to work...
Let's start again with willings.
Profile doesn't have sense for me. It's obvious. We are waiting for Oracle answer: java -Xrunhprof:cpu=samples -classpath .:ojdbc14.jar DA rank self accum count trace method 1 74.08% 74.08% 889 102 java.net.SocketInputStream.socketRead0 2 5.42% 79.50% 65 133 java.net.SocketOutputStream.socketWrite0 3 1.33% 80.83% 16 145 oracle.jdbc.ttc7.TTCItem.unmarshal 4 1.17% 82.00% 14 4 java.lang.ClassLoader.defineClass0 5 0.92% 82.92% 11 19 java.lang.ClassLoader.findBootstrapClass 6 0.83% 83.75% 10 143 oracle.jdbc.ttc7.MAREngine.unmarshalCLRforREFS 7 0.58% 84.33% 7 159 oracle.jdbc.ttc7.TTCAdapter.createNonPlsqlTTCColumnArray 8 0.58% 84.92% 7 150 oracle.jdbc.ttc7.TTC7Protocol.createDBItem 9 0.50% 85.42% 6 158 java.lang.StringBuffer.toString
I do not believe in profiling. It takes time. And during this time Oracle is working too! So, I prefer more simple way - just count milliseconds. For whole program, for database, (for sql), for cycle. See code.
And here is exact code (except sql): //========= DA.java =============
import java.io.*; import java.sql.*; import java.util.*;
public class DA { public Connection con; public Statement stmt;
static { try { Class.forName("oracle.jdbc.driver.OracleDriver"); } catch (Exception ex) { System.out.println("static driver not found: " + ex.getMessage()); } }
public void openCon(String url, String user, String password) { try { con = DriverManager.getConnection(url, user, password); stmt = con.createStatement(); } catch (Exception ex) { System.out.println(ex); } }
public void closeCon() { try { stmt.close(); con.close(); } catch (Exception ex) { System.out.println(ex); } }
public static void main(String[] args) { new DA().getData(); System.out.println("Java="+System.getProperty("java.version")); }
public PreparedStatement getPreparedStatement(String sql) throws java.sql.SQLException { return con.prepareStatement(sql); }
//////////// method itself /////////////// public Vector table; // vector of String[4]; 2000 elements // collecting_code|userid|start|end public Hashtable totaled_hrs=new Hashtable(); public void getData() { try{
double mth_hours=23*7.75; // for December 2004 String oldCode=null;
table=new Vector(); String tmp[]=null; BufferedReader br=new BufferedReader(new FileReader("load")); String line; while ((line=br.readLine())!=null){ StringTokenizer st=new StringTokenizer(line,"|"); tmp=new String[4]; tmp[0]=st.nextToken(); tmp[1]=st.nextToken(); tmp[2]=st.nextToken(); tmp[3]=st.nextToken(); table.add(tmp); } br.close(); System.out.println(table.size()); long tall,tdb,tsql,tcycle,v2,v3; tall=tdb=tsql=tcycle=0;
DA da=new DA(); ResultSet rs; Properties pr=new Properties(); pr.load(new FileInputStream("common.properties"));
//openCon("jdbc:oracle:thin:@yahoo.com:1521:users", // "scott","tiger");
da.openCon(pr.getProperty("url"), pr.getProperty("user"),pr.getProperty("password"));
double value=0,total=0; String type="";
String sql= "SELECT SUM(DECODE(TO_CHAR(e.date, 'MM'), \n"+ " '12', e.hours,0)) hours, \n"+ " d.person_type \n" + " FROM projects a, tasks b, assignments c, \n"+ " resources d, hours e, \n"+ " super_projects h, users g \n"+ " WHERE \n"+ " d.id = ? \n"+ " AND (e.date >= TO_DATE(?, 'mm/dd/yyyy')) \n"+ " AND (e.date <= TO_DATE(?, 'mm/dd/yyyy')) \n"+ " AND a.id = b.projectid \n"+ " AND b.id = c.taskid \n"+ " AND c.resourceid = d.id \n"+ " AND c.id = e.assignmentid \n"+ " AND e.approved = 'yes' \n"+ " AND a.superid = h.id \n"+ " AND h.name like 'Billed %' \n"+ " AND a.name not like 'Local %' \n"+ " GROUP BY d.person_type \n";
PreparedStatement ps = da.getPreparedStatement(sql); tall=System.currentTimeMillis();
v2=System.currentTimeMillis();
for(int i=0; i<table.size(); i++) { tmp=(String[])(table.get(i)); v3=System.currentTimeMillis(); tcycle+=v3-v2;
ps.setString(1, tmp[1]); ps.setString(2, tmp[2]); ps.setString(3, tmp[3]); rs = ps.executeQuery(); tsql+=System.currentTimeMillis()-v3; if (!rs.next()) continue; value=rs.getDouble(1); type=rs.getString(2); rs.close();
v2=System.currentTimeMillis(); tdb += v2-v3;
if (value>mth_hours&&"Billed".equals(type)) value=mth_hours;
total+=value; MyDouble d=(MyDouble)totaled_hrs.get(tmp[0]); if (d!=null) d.value+=value; else { d=new MyDouble(); d.value=value; totaled_hrs.put(tmp[0],d); } if (!(tmp[0].equals(oldCode))&&oldCode!=null) System.out.println( oldCode+" "+ ((MyDouble)totaled_hrs.get(oldCode)).value+ " "+total); oldCode=tmp[0]; }// end cycle
da.closeCon(); tall=System.currentTimeMillis()-tall;
System.out.println("Hours: "+total+ " time all="+tall+ " time db="+tdb+ " (sql="+tsql+")"+ " time cycle="+tcycle); } catch (Exception e){ e.printStackTrace(); } }
class MyDouble{public double value=0;}
} //////////// end DA.java /////////////
I hope it will post well. I'll double check it and if it will be bad I'll put it somewhere in my web site.
And here is perl code: ################# perl ######################## #!/perl/5.6/bin/perl use lib "/perl/5.6/DBD/oracle8.1.6/lib/site_perl/5.6.0/sun4-solaris"; use Oraperl;
... open connection ....
sub getData{
my(@persons)=@_; $total=0; %totaled_hrs=();
foreach $key(@persons){
@one_person=split(/:/,$key);
$sql=...;
$db_list=&ora_open($lda,$sql) || die "\nCan't open cursor 1: $ora_errstr\n";
($time, $type)=&ora_fetch($db_list);
if ($time>178.25) { if ( "Billed" eq $type ) { $time=178.25; }}
$totaled_hrs{$one_person[0]}+=$time; $total+=$time;
} ############# end of perl ################
Result of java is Hours: 154930.95 time all=5436908 time db=5297897 (sql=5434009) time cycle=243005 Java=1.4.2_01
Which means 90 minutes for whole run. Where 88 minutes for Oracle waiting. Total Java expenses 4.5 minutes. (Please, don't be too strict to numbers. I just cut&paste them. Milliseconds are a little big time period. Be indulgent to this quick and not precision way.)
perl result is: Hours: 154930.95 for 1487 seconds. Which is 25 minutes today. Which is 3.6 times faster.
So, I'm open to any suggestions how to improve my Java code or how to force managers to use Java in this project.
Alex Kizub.
bugbear - 01 Feb 2005 10:50 GMT > I do not believe in profiling. It takes time. You ask experts for advice, and then refuse to take it. Your choice.
BugBear
akizub@yahoo.com - 01 Feb 2005 13:09 GMT > > I do not believe in profiling. It takes time.
>You ask experts for advice, and then refuse to take it. Your choice.
> BugBear Read the post! I did profile! And it was as I expected. Program itself is so simple that do not require profile. It's obvious that problem is somewhere in Oracle connection. Which means that it doesn't matter how good is your program - connection kills it. Very pity.
I gave up and left this application on perl.
Alex Kizub.
Luke Webber - 01 Feb 2005 21:49 GMT >>>I do not believe in profiling. It takes time. > [quoted text clipped - 10 lines] > > I gave up and left this application on perl. It seems clear that you're correct about the time being spent in the JDBC connection logic, but I'm wondering just which part of the JDBC code is taking all the time. The figures aren't fine-grained enough.
I have one off-the-wall suggestion. Why not try removing the "rs.close()" statement and see how it goes? I have a niggling feeling that it might be making an extra round trip to the database server for some reason.
Cheers, Luke
Thomas Kellerer - 01 Feb 2005 22:09 GMT Luke Webber wrote on 01.02.2005 22:49:
> I have one off-the-wall suggestion. Why not try removing the > "rs.close()" statement and see how it goes? I have a niggling feeling > that it might be making an extra round trip to the database server for > some reason. I don't think that's a good idea. That will leave him with a lot of unclosed cursors...
Thomas
Robert Sundström - 02 Feb 2005 16:37 GMT > Luke Webber wrote on 01.02.2005 22:49: >> I have one off-the-wall suggestion. Why not try removing the [quoted text clipped - 4 lines] > I don't think that's a good idea. That will leave him with a lot of > unclosed cursors... Not necessarily.
If I understand things, the original poster has an iteration where he executes a PreparedStatement over and over again. Any open ResultSet objects originating from that PreparedStatement should, if the JDBC driver complies with the JDBC spec, be automatically closed at the next execute.
However, it would be most strange if the 2000 extra round trips to the server can cause the delay. I would say that the server, for some reason optimizes the SQL-statements differently. It could be that the Oracle-Perl driver internally flips some magic option or compilation directive which in turn makes the optimizer make better or less good decisions. The original poster is probably better off asking this question at an Oracle group.
 Signature Robert Sundström, Mimer SQL Development Mimer Information Technology AB, http://www.mimer.com Validate your SQL statements/procedures at http://developer.mimer.com/validator
Dimitri Maziuk - 02 Feb 2005 18:45 GMT Thomas Kellerer sez:
> Luke Webber wrote on 01.02.2005 22:49: >> I have one off-the-wall suggestion. Why not try removing the [quoted text clipped - 4 lines] > I don't think that's a good idea. That will leave him with a lot of > unclosed cursors... Nope. Oracle doesn't close the cursor on rs.close(), it closes on query.close() (by default, anyway).
Dima
 Signature The wombat is a mixture of chalk and clay used for respiration. -- MegaHal
steve - 04 Feb 2005 21:34 GMT >>>> I do not believe in profiling. It takes time. >> [quoted text clipped - 22 lines] > Cheers, > Luke he should not be calling an sql statement 2000 times, when he can do it in the server, and return a cursor.
and if he is so stubbon to do a callout 2000 times he can bulk it up to 1 call, execute & then close. not callout 2,000 times , closing each time.
the code is broken, implementation wise, and it can be profiled to the end of time, but it is not going to be able to fix it like that.
steve
Chris Smith - 05 Feb 2005 03:18 GMT > he should not be calling an sql statement 2000 times, when he can do it in > the server, and return a cursor. Clearly this is only true in a limited set of situations. Neither you nor I know enough about Alex's application to know if it's true or not. Nevertheless, it's not at all relevant to the question of why the performance between Java and Perl might differ.
It appears the point of your post is that you feel smart. I'm glad you feel smart. Self-esteem is always a good thing. When you follow up with incorrect advice, it makes you seem even smarter. When you accentuate your self-esteem with random cursing in public to show your disdain for others, it makes you seem like a really nice guy.
 Signature www.designacourse.com The Easiest Way To Train Anyone... Anywhere.
Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
steve - 04 Feb 2005 21:38 GMT >>>> I do not believe in profiling. It takes time. >> [quoted text clipped - 22 lines] > Cheers, > Luke it will stop ( well actually sql crash , and it would kill any other program on the server that tries to open a cursor ) after 300 cursors are left open. not the way to run on a production system.
steve
Chris Smith - 05 Feb 2005 03:09 GMT > it will stop ( well actually sql crash , and it would kill any other program > on the server that tries to open a cursor ) after 300 cursors are left open. > not the way to run on a production system. Steve, you're a little late here, but just to clarify: that's not true. Omitting rs.close() in the code mentioned here will not leave open cursors anywhere, because the cursor will be closed when the next loop iteration calls PreparedStatement.execute(). The last cursor is closed when the connection is closed.
It is possible to avoid this behavior, using the prepareStatement overload (new in 1.4) that takes an additional parameter on result set holdability. Since this is not used in the OP's code, your concern is not valid. It turns out that there's very rarely a reason to explicitly close ResultSet instances, unless you went out of your way to keep them open in the first place.
In any case, Alex already has a solution to his problem.
 Signature www.designacourse.com The Easiest Way To Train Anyone... Anywhere.
Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
Lee Fesperman - 05 Feb 2005 03:27 GMT > > it will stop ( well actually sql crash , and it would kill any other program > > on the server that tries to open a cursor ) after 300 cursors are left open. [quoted text clipped - 5 lines] > iteration calls PreparedStatement.execute(). The last cursor is closed > when the connection is closed. That's what the spec says, but I think that closing resultsets is needed for some backends (Oracle). Also, you should close the prepared statement instead of waiting for the connection to do the job. IOW, the best practice is always close resultsets, statements and connections and do it as soon as you don't need them anymore. For a decent backend, this should not be more costly and will free up resources on the server ... to everyone's benefit.
 Signature Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com) ============================================================== * The Ultimate DBMS is here! * FirstSQL/J Object/Relational DBMS (http://www.firstsql.com)
Chris Smith - 05 Feb 2005 04:43 GMT > That's what the spec says, but I think that closing resultsets is needed for some > backends (Oracle). You're saying that Oracle is providing terribly broken JDBC drivers that cause a normal and correct JDBC application to cause severe problems on their server, and hasn't fixed this in nearly 10 years? That would really be rather astonishing, quite frankly. Any reference to provide evidence for that?
 Signature www.designacourse.com The Easiest Way To Train Anyone... Anywhere.
Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
Joe Weinstein - 05 Feb 2005 04:51 GMT >>That's what the spec says, but I think that closing resultsets is needed for some >>backends (Oracle). [quoted text clipped - 4 lines] > really be rather astonishing, quite frankly. Any reference to provide > evidence for that? It's a function of DBMS-side cursors which are associated with open statements and open result sets. It is common, correct JDBC hygiene to close such objects ASAP, and for Oracle it is more important than for some DBMSes. Oracle permits a fixed (configurable) number of cursors that a client can have open at one time. An application that caches or ignores too many open statements or result sets can get errord if it exceeds the DBMS limit. It's not a JDBC issue as such.
Joe Weinstein at BEA
Chris Smith - 05 Feb 2005 05:31 GMT > It's a function of DBMS-side cursors which are associated with open statements > and open result sets. It is common, correct JDBC hygiene to close such objects > ASAP, and for Oracle it is more important than for some DBMSes. Oracle permits > a fixed (configurable) number of cursors that a client can have open at one > time. An application that caches or ignores too many open statements or result > sets can get errord if it exceeds the DBMS limit. It's not a JDBC issue as such. Apparently you haven't been reading the thread. Oracle's JDBC driver should not be holding a ResultSet open across a Connection.close() -- nor a commit() unless a non-default holdability was set using JDBC3's new overloads of createStatement and prepareStatement. Nor should it survive a Statement.execute() or getMoreResults() (the version with no parameters). If the Oracle driver is producing ResultSet instances that sutvive these things, then it is in violation of the spec, including (but not limited to) sections 10.1, 14.1.3, 14.2.5, Oracle's driver is broken, and it's broken in a particularly gruesome way.
The spec (somewhat confusingly, and for no good reason that I can see) allows for greater leeway for Statement.close() -- requiring the driver to immediately invalidate and "close" the ResultSet, but permitting it to hold off on releasing resources until the next garbage collection. That's not a particularly useful allowance, and I imagine that there are probably no drivers that take advantage of it but still comply with the spec. It appears that the author of the specification does not actually understand garbage collection in Java. That theory is confirmed when section 14.2.5 specifies that calling ResultSet.close() makes the ResultSet object immediately available for garbage collection -- a ludicrous requirement that invalidates the entirety of the Java security model, and requires a heavily modified virtual machine to actually implement correctly (luckily, no one actually takes this requirement of the spec seriously).
Clearly that is in fact a JDBC issue, since it's about whether Oracle's driver complies certain requirements of the JDBC specification or not. If it is confirmed that this is a problem with Oracle's JDBC driver, then apparently it will become necessary to make calls to ResultSet.close() in the middle of a tight loop of PreparedStatement's execute() calls. That would then be classified as a work-around for a serious bug, *not* as good general practice for JDBC.
 Signature www.designacourse.com The Easiest Way To Train Anyone... Anywhere.
Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
Joe Weinstein - 05 Feb 2005 05:55 GMT > Apparently you haven't been reading the thread. you're right, sorry.
Oracle's JDBC driver
> should not be holding a ResultSet open across a Connection.close() -- > nor a commit() unless a non-default holdability was set using JDBC3's > new overloads of createStatement and prepareStatement. I'll catch up on the thread. Is that happening?
> Nor should it > survive a Statement.execute() or getMoreResults() (the version with no > parameters). If the Oracle driver is producing ResultSet instances that > survive these things, then it is in violation of the spec, including > (but not limited to) sections 10.1, 14.1.3, 14.2.5, Oracle's driver is > broken, and it's broken in a particularly gruesome way. I agree. When last I checked, the result sets that came from standard calls, such as executeQuery() *were* closed properly when the statement was closed or re-executed, but those result sets that arrived via non-standard JDBC, such as via output parameters of procedures, via their non-standard CallableStatement.getResultSet( int paramIndex) were not closed. A bug indeed, if it's still true. In fact the driver used to allow a user to execute a stored procedure via executeQuery() which must return a result set. The result set was some odd bogus result set that didn't relate to the procedure output.
> The spec (somewhat confusingly, and for no good reason that I can see) > allows for greater leeway for Statement.close() -- requiring the driver [quoted text clipped - 10 lines] > implement correctly (luckily, no one actually takes this requirement of > the spec seriously). Yep, silly. I still have a reference to it. *Try* gc()'ing it... Even if it is closed and the reference is gone, until getMoreResults() is called, one could conceivably call getResultSet() again and expect to get the same already-closed result set, so the statement has to keep a reference for a while too.
> Clearly that is in fact a JDBC issue, since it's about whether Oracle's > driver complies certain requirements of the JDBC specification or not. [quoted text clipped - 3 lines] > execute() calls. That would then be classified as a work-around for a > serious bug, *not* as good general practice for JDBC. I always close everything asap... Chaqu'un a son gout... but yes, I wasn't reading the thread. Joe Weinstein at BEA
Chris Smith - 05 Feb 2005 14:09 GMT > Oracle's JDBC driver > > should not be holding a ResultSet open across a Connection.close() -- > > nor a commit() unless a non-default holdability was set using JDBC3's > > new overloads of createStatement and prepareStatement. > > I'll catch up on the thread. Is that happening? I don't know. Lee suggested that it is happening. I hope he's wrong.
> Yep, silly. I still have a reference to it. *Try* gc()'ing it... > Even if it is closed and the reference is gone, until getMoreResults() > is called, one could conceivably call getResultSet() again and expect to > get the same already-closed result set, so the statement has to keep a > reference for a while too. I agree with all but the getResultSet() piece. getResultSet() is only guaranteed to work once per result. The very first time you call it for a given result, the Statement is entirely within its rights to throw away any references to that ResultSet. After that, it can throw SQLException instead of returning a ResultSet.
> I always close everything asap... Chaqu'un a son gout... but yes, I wasn't > reading the thread. I certainly would not argue for leaving things open indefinitely. Specifically, though, we're discussing whether it's reasonable to omit a call to ResultSet.close() in the middle of a tight loop of calls to PreparedStatement.executeQuery(). As far as I'm concerned, that IS basically closing things ASAP; the next call to executeQuery is going to happen immediately, and it will close the ResultSet. Unless Lee is right about Oracle's driver bug...
 Signature www.designacourse.com The Easiest Way To Train Anyone... Anywhere.
Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
Dimitri Maziuk - 05 Feb 2005 18:51 GMT Chris Smith sez:
>> Oracle's JDBC driver >> > should not be holding a ResultSet open across a Connection.close() -- [quoted text clipped - 4 lines] > > I don't know. Lee suggested that it is happening. I hope he's wrong. Yep, he is. I don't know about result sets returned from stored procedures, but regular ones get closed as per specs (at least in 8i and 10g).
IME it's statement.close() that releases server-side cursor; you want to explicitly close your statements ASAP, if you leave that to gc you're likely to run out of cursors. Moreover, it seems that gc'ing unclosed statements does not always close server side of the connection, sometimes it sits there hours waiting to time out.
>> I always close everything asap... Chaqu'un a son gout... but yes, I wasn't >> reading the thread. [quoted text clipped - 5 lines] > basically closing things ASAP; the next call to executeQuery is going to > happen immediately, and it will close the ResultSet. I always close the statements and usually close associated result set just before stat.close(), just to be sure. I never close the result set in a middle of a loop (tight or not) and so far I haven't run into any problems.
... Unless Lee is
> right about Oracle's driver bug... There's 2 thin drivers in 10g: new ...OracleDriver and old ...driver.OracleDriver. The old one is retained for compatibility backwards, all bugfixes and new features go into new one. Plus there's not-thin driver(s?), RowSet implementation jar, etc. There may be that Lee's bug is in one or two of them, who knows...
Dima
 Signature Riding roughshod over some little used trifle like the English language is not a big deal to an important technology innovator like Microsoft. They did just that by naming a major project dot-Net (".Net"). Before that, a period followed by a capital letter was used to mark a sentence boundary. --T. Gottfried, RISKS 21.91
Lee Fesperman - 05 Feb 2005 21:05 GMT > > That's what the spec says, but I think that closing resultsets is needed for some > > backends (Oracle). [quoted text clipped - 4 lines] > really be rather astonishing, quite frankly. Any reference to provide > evidence for that? Don't be naive. Do you really think Oracle gives a fig about compatibility with specs? SQL Server is much worse, and DB2 is not so hot. These are the 800 pound gorillas. If you want to use them, you have to put up with their eccentricities. After that there is a fairly large number of other JDBC drivers, including more than one for some DBMSs and then there are different versions. Add all the ODBC drivers accessible through JDBC-ODBC bridges, you'll find that compatibility is all over the map.
Remember that the next time you recommend a feature like resultset holdability (new in JDBC 3). The rate of adoption is slow and spotty. Some will never support new features.
At least with JDBC, much of this can be covered with defensive programming. Have you ever written a JDBC application intended to run with a variety of drivers and backends? I have (several in fact). It is possible to get reasonable coverage if you're careful. The situation with SQL is much worse. Everyone has their own proprietary version of the language.
 Signature Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com) ============================================================== * The Ultimate DBMS is here! * FirstSQL/J Object/Relational DBMS (http://www.firstsql.com)
Dimitri Maziuk - 06 Feb 2005 18:03 GMT Lee Fesperman sez: ...
> At least with JDBC, much of this can be covered with defensive programming. Have you > ever written a JDBC application intended to run with a variety of drivers and backends? > I have (several in fact). It is possible to get reasonable coverage if you're careful. > The situation with SQL is much worse. Everyone has their own proprietary version of the > language. I love how Oracle's thin driver doesn't have DatabaseMetadata.getDatabaseXXXVersion(). So that you can't do if( version >= 10 ) stat.executeUpdate( "drop table foo purge" ); else stat.executeUpdate( "drop table foo" ); without a pattern match on the product name string. Hatess 'em, we doess...
Dima
 Signature Backwards compatibility is either a pun or an oxymoron. -- PGN
Lee Fesperman - 06 Feb 2005 19:39 GMT > Lee Fesperman sez: > ... [quoted text clipped - 10 lines] > without a pattern match on the product name string. > Hatess 'em, we doess... Interesting. What does "drop table foo purge" mean?
 Signature Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com) ============================================================== * The Ultimate DBMS is here! * FirstSQL/J Object/Relational DBMS (http://www.firstsql.com)
Dimitri Maziuk - 07 Feb 2005 16:55 GMT Lee Fesperman sez: ...
> Interesting. What does "drop table foo purge" mean? It doesn't delete tables anymore, it renames them to RECYCLEQWJQ#(*$@!#JR.BIN instead. "Purge" means "don't clutter my tablespace with junk, I really mean to delete the fscking thing and I know what I'm doing".
Dima
 Signature Tlaloc: What was Elrond's second name? Gruber: Hubbard -- <ahbou=3C69EB63.A7C431F4@last.com>
Lee Fesperman - 07 Feb 2005 20:24 GMT > Lee Fesperman sez: > ... [quoted text clipped - 4 lines] > "don't clutter my tablespace with junk, I really mean > to delete the fscking thing and I know what I'm doing". Thanks for the explanation. The 800 pound gorilla at work again ;^)
 Signature Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com) ============================================================== * The Ultimate DBMS is here! * FirstSQL/J Object/Relational DBMS (http://www.firstsql.com)
Robert Klemme - 01 Feb 2005 13:54 GMT > Luke: > >we're trying to help [quoted text clipped - 22 lines] > oracle.jdbc.ttc7.TTC7Protocol.createDBItem > 9 0.50% 85.42% 6 158 java.lang.StringBuffer.toString Did you feed that into an analysis tool like HPJmeter? Maybe there's something different going on on the network.
Also, for the db it might be more efficient if you include the data you draw from variable "table" and execute a slightly modified SQL just once.
robert
> I do not believe in profiling. It takes time. > And during this time Oracle is working too! [quoted text clipped - 235 lines] > > Alex Kizub. akizub@yahoo.com - 01 Feb 2005 14:48 GMT >Did you feed that into an analysis tool like HPJmeter? I used HPjmeter-1.6. It's a little old. Latest version is for 2003. And supports JAVA PROFILE 1.0 when latest java produces JAVA PROFILE 1.0.1. Nevertheless you can see original text file which is obvious enough.
>Maybe there's something different going on on the network. Problem is that both perl and java are working in the same environment, the same time, the same database and the same sql. And result is always the same. perl at least twice faster. If java runs 90 minutes then perl runs 25. If java runs 43 minyes then perl 17. And so on.
>Also, for the db it might be more efficient if you include the data you draw from variable "table" and execute a slightly modified SQL just once.
Of course I can modify SQL. But I have to do it for both langauges. So this is not the point for difference. And, I assume, that 2000 SQLs are good enough for statistic's measure.
Disappointing point is that if I use simple SQL like "select 1 from dual' then both languages have the same result. Which means for me that overhead charges are the same. When I change results (like increase key numbers for hastable, make them unique = 2000 all different, but for the same simple query) then java is faster. Which means calculation itself are better in java.
But database connection kills everything. I had another experience with IBM original JDBC connection to mainframe DB2. select for 1,000,000 items table was about 5 minutes. But I thought that IBM forced programers to use COBOL not Java. And gave up for it too. My Java world shrinks so rapidly....
:((((((((((((((( Alex Kizub.
dar7yl - 01 Feb 2005 22:48 GMT Alex:
I think I can see where a lot of your time is being spent - in the sql query and the subsequent returning of the resultset.
If, as you say, the query is generated the same in java as it is in perl, then the only differences would be the transmission of the resultset, as the sql server would be performing the same operation (nearly enough) in both cases.
If your jdbc driver supports it, turn on tracing/profiling. The MySQL driver has this feature which you turn on by adding "?&profileSql=true" to the connect string. This is a great feature, which shows the actual expanded query, the server execution time, and the transmission time.
I assume that the query returns only one result. It would be better programming practice to surround the logic with if (rs.next()) { // logic goes here } instead of performing the continue on the negative case. (bad style). Even better would be to make that a while, just in case the query returns more than one result.
Perhaps, you could optimize the query to return the whole list in one call. A way to do that would be to create a temporary table from your getData() table and then do a join in your sql query, thus returning a dataset with more than one row. This would take a great deal of load off the server, as it would only have to perform the scan once (albeit a more complex query).
regards, Dar7yl
> Luke: >>we're trying to help [quoted text clipped - 262 lines] > > Alex Kizub. akizub@yahoo.com - 02 Feb 2005 02:10 GMT Dar7yl: Thank for answer.
>If your jdbc driver supports it, turn on tracing/profiling. I use jodbc14.jar from Oracle. Don't know how to force it to trace. Tomorrow will have a look on oracle.com. But what is the diffence where time is spent? I still have to pay the price.
> if (rs.next()) > { > // logic goes here > } Of course I usually do this. You can see it from previous code. I only tried to simplify the code and reduce number of getMilliseconds calls.
>Perhaps, you could optimize the query to return the whole list in one call. Of course I do it in real project. And I can do it in both Java or perl. But then quiry eats so much power of Oracle that DBAs call me and ask to stop it. Did you ever had such experience when your query runs for hour and DBA call you and ask what is that query which combines 9 tables and hold whole system for an hour? What can I answer for such DBA? That this is your database structure and your relations? And there is no other way to obtain data from your imparctical design? And people like this design and support this system... So, I ask no questions. I ask them here :))))
Anyway. It seems for me more and more that difference is in OCI driver for perl. I'll try to play with it but will take time. And anyway again it's rear approach. Most applications use thin driver. And when it three times slower then OCI... too bad.
Thanks anyway. I already gave up. But it was good experience for me. I know a lot of new stuff and will know even more (with this OCI I hope).
And project remains on perl. But who cares! perl is much better then COBOL. Another of my previous projects is still on COBOL :(((
Alex Kizub.
Tom Dyess - 02 Feb 2005 03:03 GMT > Dar7yl: > Thank for answer. [quoted text clipped - 44 lines] > > Alex Kizub. Can't you exp the schema/tables and imp it onto another server and tune it without bothering anyone else? Don't you have a development box at work? There is something really wrong going on and you're gonna have to tinker with it. One of us could probably find it after a few minutes if we could connect to your environment, but in newsgroups, its just conjecture and theory. Try putting a /*+ RULE */ hint in your SQL (after the first select) and make sure your indexes are in place. You're connected to the proper instance - i.e. there aren't two instances on the same box and you're connecting to one via perl and the other via Java are you?
 Signature Tom Dyess OraclePower.com
akizub@yahoo.com - 02 Feb 2005 13:12 GMT >Can't you exp the schema/tables and imp it onto another server and tune it without bothering anyone else? No. On devl it works fine.
> Don't you have a development box at work? Our DBAs can't replicate situation in devl. It exists only in prod.
>There is something really wrong going on I totally agree.
> and you're gonna have to tinker with it. And that's what I'm doing.
> One of us could probably find it after a few minutes if we could connect to your environment, It's not even mine.
>its just conjecture and theory. And that's what I'm talking about.
> Try putting a /*+ RULE */ hint in your SQL (after the first select) and make sure your indexes are in place. This is not about tune or performance. This is about comparisond two languages. So, everything should be the same. Bad or good.
>You're connected to the proper instance - i.e. there aren't two instances on the same box and you're connecting to one via perl and the other via Java are you?
Are you kidding? Everything is the same.
Sorry. Couldn't resist to answer to guy from OraclePower.com Even topic is closed. Sincerely, Alex Kizub.
Tom Dyess - 02 Feb 2005 13:27 GMT > >Can't you exp the schema/tables and imp it onto another server and > tune it [quoted text clipped - 33 lines] > Sincerely, > Alex Kizub. Yea, I just don't like seeing it die. I like to solve problems but looks like this one is taking a dirt nap. I thought the objective was to make the Java version run at a comperable speed to Perl as opposed to more of an academic nature. To do that, it requires more than your environment is willing to let you do. Sounds like your DBA won't even let you work on it much less give you explain plans, so you're kinda feeling in the dark. Oh well, it was fun anyway.
Tom Dyess OraclePower.com
Alex - 02 Feb 2005 21:10 GMT My apology. I became victim of circumstances and my own ambitions. That's production database and performance varies too much. Right now I did a lot of other statistics and discovered that in average SQL time varies 100 times. So, when I saw that perl was faster (and it was a little faster because used OCI driver) and after 3 to 5 runs (remember, they take about hour) I changed SQL from plain one to prepared statement. So, when perl sent new SQL every time Java used parameters in predefined one. And it became even slower. I was upset and started this topic.
Now DBA (other, not ours) did a little research and discovered that Explanation Plan which Oracle uses for prepared statement is totally different then for plain text SQL. And based on whatever it was based this plan in reality was much worse that plan for the same SQL but with values except parameters.
So, I changed SQL back from prepared to generated for each run (just as perl did) and got approximately the same results for both languages. Both, again, were different time to time at least 10 times but, at least similar. Sometime j |
|