Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / Databases / August 2003

Tip: Looking for answers? Try searching our database.

ResultSet size

Thread view: 
Halldór Ísak Gylfason - 27 Aug 2003 15:46 GMT
I am currently wondering what is the default behaviour of jdbc
database drivers with regard to if they return a ResultSet that is
already fully populated with data , or if they use some lazy loading
mechanism.

Basically I have a really large table that will not fit in memory, but
I still need to do a "SELECT * FROM LARGE_TABLE" query and process all
the rows. The question is if I must myself implement the logic to
avoid OutOfMemoryError or if I can rely on correct behaviour of the
JDBC driver.

The Statement interface includes a method setFetchSize(int nrRows)
that seems to imply that I can set the number of rows that are
fetched, but in order for that to work the ResultSet must of course
discard the existent rows when fetching new rows.

I looked at the implementation of the Statement interface in
Hypersonic (jdbcStatement.java) and interestingly enough the parameter
is ignored and everything is fetched, which makes Hypersonic not
usable in situations like that.
Joseph Weinstein - 27 Aug 2003 16:33 GMT
> I am currently wondering what is the default behaviour of jdbc
> database drivers with regard to if they return a ResultSet that is
[quoted text clipped - 16 lines]
> is ignored and everything is fetched, which makes Hypersonic not
> usable in situations like that.

Sorry. Most drivers do a pre-read of 1-N rows, and repeat when needed,
but won't discard rows already read. The setFetchSize() is just a hint at
how many rows to pre-fetch at a time. It has nothing to do with how many
rows in total to fetch or keep. Some drivers may discard row data once
that row has been passed, for result sets that are forward-only. Some don't.
  You are suffering the fundamental problem of using the DBMS as a dumb
file system. The number one performance tenet of DBMS application architecture
is to never ship vast quantities of raw data from the DBMS to a client for
processing. The idea is to use stored procedures to do the bulk processing
where the data is.
  If you can't do this, for whatever reason, you might consider making your
SQL fancier, to fetch the data in batches you define (one batch per result set),
so you can control whether a result set stays in memory etc.
Good luck,
Joe Weinstein at BEA
Halldór Ísak Gylfason - 28 Aug 2003 10:41 GMT
One of the reasons I was asking this is because I have been
experimenting with the JMS Message Queue implementation of JBoss. I
have found out that that implementation is not good at all, and
unusable with high message loads. To be fair however JBoss people
acknowledge that (at least some of them) and they are rewriting it for
4.0

One of the things that the JBoss messaging subsystem does  when
booting up, is the following query to the database (when using a
PersitanceManager that stores messages in the database):

"SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE DESTINATION=?"

In other words it does a query for all messages on a single queue. If
that queue is big, there will obviously be memory problems like I was
pointing out in my message. The code that does the queue restoring in
JBoss in case you're interested is in the
org.jboss.mq.pm.jdbc2.PersistenceManager and the function is
restoreQueue:

  /////////////////////////////////////////////////////////////////////////////////
  //
  // Message Recovery
  //
  /////////////////////////////////////////////////////////////////////////////////
  synchronized public void restoreQueue(JMSDestination jmsDest,
SpyDestination dest) throws javax.jms.JMSException
  {
     if (jmsDest == null)
        throw new IllegalArgumentException("Must supply non null
JMSDestination to restoreQueue");
     if (dest == null)
        throw new IllegalArgumentException("Must supply non null
SpyDestination to restoreQueue");

     TransactionManagerStrategy tms = new
TransactionManagerStrategy();
     tms.startTX();
     Connection c = null;
     PreparedStatement stmt = null;
     ResultSet rs = null;
     boolean threadWasInterrupted =
Thread.currentThread().interrupted();
     try
     {

        c = this.getConnection();
        stmt = c.prepareStatement(SELECT_MESSAGES_IN_DEST);
        stmt.setString(1, dest.toString());

        rs = stmt.executeQuery();
        int counter=0;
        while (rs.next())
        {
           SpyMessage message = extractMessage(rs);
           // The durable subscription is not serialized
           if (dest instanceof SpyTopic)
              message.header.durableSubscriberID =
((SpyTopic)dest).getDurableSubscriptionID();
           jmsDest.restoreMessage(message);
           counter++;
        }
       
        log.debug("Restored "+counter+" message(s) to: "+dest);
     }
     catch (IOException e)
     {
        tms.setRollbackOnly();
        throw new SpyJMSException("Could not restore messages to
destination : " + dest.toString(), e);
     }
     catch (SQLException e)
     {
        tms.setRollbackOnly();
        throw new SpyJMSException("Could not restore messages to
destination : " + dest.toString(), e);
     }
     finally
     {
        try
        {
           rs.close();
        }
        catch (Throwable ignore)
        {
        }
        try
        {
           stmt.close();
        }
        catch (Throwable ignore)
        {
        }
        try
        {
           c.close();
        }
        catch (Throwable ignore)
        {
        }
        tms.endTX();

        // Restore the interrupted state of the thread
        if( threadWasInterrupted )
           Thread.currentThread().interrupt();
     }

  }


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.