Which data structure to use for large volumes of data?

Thread view:

Arun Shastry - 30 Dec 2003 01:57 GMT

Hi,

We have an application that brings thousands of records from an Oracle
database (50,000+). The application is developed in EJB, deployed on JBoss.
We're currently using arraylists in a stateless session bean to contain the
recordset and keep bumping into JVM heapsize problem.

Can anyone suggest a better data structure to handle large volumes of data?
Is JVM heapsize configurable dynamically programatically (i.e. we estimate
the size of the records we are about to retrieve from Oracle and adjust the
JVM heapsize prior to fetching the data...)?

Thanks,
Arun

Reply to this Message

Johann Burkard - 30 Dec 2003 02:18 GMT

> We have an application that brings thousands of records from an Oracle
> database (50,000+). The application is developed in EJB, deployed on JBoss.
> We're currently using arraylists in a stateless session bean to contain the
> recordset and keep bumping into JVM heapsize problem.

Why would you need to cache all the records? Find out which ones are
used the most and cache only these. [1] [2] Or cache the ones that are
shown first to the client. People never dive that much into long lists.

> Can anyone suggest a better data structure to handle large volumes of data?

Yeah, it's called RAM.

> Is JVM heapsize configurable dynamically programatically (i.e. we estimate
> the size of the records we are about to retrieve from Oracle and adjust the
> JVM heapsize prior to fetching the data...)?

I don't think so. I'd rather try to find out what data you have, why you
think you need to cache it, how the data is accessed and what the
constraints are.

[1] http://jakarta.apache.org/turbine/jcs/index.html
[2] http://jakarta.apache.org/commons/collections.html has a MRU Map IIRC.

Johann

Signature

hmm
<test>
blödmann !
</test> ("scheffe" in <bqjc63$pjv$01$1@news.t-online.com>)

Reply to this Message

Soren Kuula - 02 Jan 2004 14:17 GMT

Hi,

> Hi,
>
[quoted text clipped - 4 lines]
>
> Can anyone suggest a better data structure to handle large volumes of data?

How about this :

Instead of passing complete sets of data between the various components
of your program, instead pass objects that can iterate over the data.

A ResultSet can do that, but ResultSets are not nice things to pass
around in an architecture. I suppose that somewhere you make beans out
of rows. So write a class that implements an Iterator-like interface,
and has a reference to a ResultSet and inherits from / refers to the
code that makes the beans. Now you have access to all of your results
without actually needing to load them all at the same time.

Of course, if you actually need to USE all the results at the same time
(but who would ?) then this won't help.

Soren

Signature

Fjern de 4 bogstaver i min mailadresse som er indsat for at hindre s...
Remove the 4 letter word meaning "junk mail" in my mail address.

Reply to this Message

Carlo Pellegrini - 12 Jan 2004 08:53 GMT

> Hi,
>
> We have an application that brings thousands of records from an Oracle
> database (50,000+). The application is developed in EJB, deployed on JBoss.
> We're currently using arraylists in a stateless session bean to contain the
> recordset and keep bumping into JVM heapsize problem.

Well, it's one of the problems addressed by the Entity
Beans...
Build an entity bean wich maps a row of your recordset, and
store
their reference in your arrayList (hopefully with a lazy
loading strategy).
The container will activate/passivate them as necessary.

CarloP

Reply to this Message

Capitan Mutanda - 14 Jan 2004 07:43 GMT

> > Hi,
> >
[quoted text clipped - 11 lines]
> their reference in your arrayList (hopefully with a lazy
> loading strategy).

Please elaborate.
I was thinkiny of a stateless session that is a window on the
cursor of the DB

> The container will activate/passivate them as necessary.
>
> CarloP

Reply to this Message

Which data structure to use for large volumes of data?

Free Magazines