Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / October 2007

Tip: Looking for answers? Try searching our database.

Performance Issues in Random Access Files

Thread view: 
gwlucas@sonalysts.com - 09 Oct 2007 17:31 GMT
I have an application where I need to read data from random posiitions
within file. In this case performance is VERY important. The problem I
am running into is that the RandomAccessFile class appears to have
serious performance issues. Based on a bit of experimentation, I
suspect that it doesn't implement any kind of intrinsic buffering.

Anyway, I can think of five or six ways of working around this
problem, but I can't believe that there isn't a standard solution. As
a rule, I like to stick to "recognized practices of the community"
even when it would be more fun to "roll my own." So... is there a
canonical solution?  Would somebody be able to point me in the right
direction?

Background

I have a file containing blocks of data that need to be accessed at
random. I never know which block I am going to need to read next but,
within each block, the data is read in sequence. The pattern of access
is somewhat similar to an old fashioned ISAM file. Thus I would like
to set the file pointer to the beginning of the block, read some
integers, read some doubles, etc, all in sequence. Later, I would jump
to another file position and do the same. Normally, I would accomplish
these sequential reads with a BufferedInputStream. But since I do have
the random access component, that doesn't seem to be an option
(apparently, you cant wrap a BufferedInputStream around a random
access file).

Thanks in advance for your help.

Gary
gwlucas@sonalysts.com - 09 Oct 2007 19:01 GMT
On Oct 9, 12:31 pm, gwlu...@sonalysts.com wrote:
> I have an application where I need to read data from random posiitions
> within file. In this case performance is VERY important. [snip]

An addenda...  Strictly for testing purposes, I have implemented a
wrapper around RandomAccessFile following a suggestion posted here by
Tom Anderson on 10 Oct 2002 which
makes a class that looks like an InputStream but makes pass-through
calls to RandomAccessFile.  It works, but definitately falls into the
kludge category.

I was wondering whether the java.nio classes might be a solution here.
I've never used them and just assumed that they were related to
network I/O, but there does appear
to be some stuff related to FileChannels that might be relevant.
Frankly, the whole java.nio API looks rather unfathomable.

Thanks again.

Gary
Esmond Pitt - 12 Oct 2007 04:10 GMT
> I was wondering whether the java.nio classes might be a solution here.

They are indeed. What you want is FileChannel.map(), then you just deal
directly with a MappedByteBuffer. This nly works reasonably for fixed
length files of course ... but that's much the same for any file-mapping
API.
Roedy Green - 11 Oct 2007 02:38 GMT
>I have an application where I need to read data from random posiitions
>within file. In this case performance is VERY important. The problem I
>am running into is that the RandomAccessFile class appears to have
>serious performance issues. Based on a bit of experimentation, I
>suspect that it doesn't implement any kind of intrinsic buffering.

see http://mindprod.com/jgloss/nio.html

I would try implementing this with nio. If the file in not too
enormous, you can use the memory mapping features to tap into the
system caching.
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

gwlucas@sonalysts.com - 12 Oct 2007 14:46 GMT
Roedy and Esmond,

Thank you both for your help (Roedy, I am a long time admirer of your
contributions to the Java community).

I plan on looking into Java NIO. Roedy's web site provides a link to a
pretty extensive tutorial.

In the mean time, I tried some experiments using Tom Andersion's 2002
suggestion (which pre-dates NIO) of wrapping RandomAccessFile in a
container class that allows it to be wrapped in a BufferedInputStream.

Just using a quick-and-dirty approach, one that makes very wasteful
use of object creation, I was achieved a factor of 9 improvement in
speed. It would be interesting to see if it would gets better if I
took a careful approach. Anyway, it turns out that RandomAccessFile is
a pretty lousey implementation...  a big step back from the C language
standard i/o implementation of fread/fwrite that was created in the
1970's.

Gary

On Oct 10, 9:38 pm, Roedy Green <see_webs...@mindprod.com.invalid>
wrote:
> On Tue, 09 Oct 2007 09:31:20 -0700, gwlu...@sonalysts.com wrote,
> quoted or indirectly quoted someone who said :
[quoted text clipped - 13 lines]
> Roedy Green Canadian Mind Products
> The Java Glossaryhttp://mindprod.com
Esmond Pitt - 15 Oct 2007 01:33 GMT
Anyway, it turns out that RandomAccessFile is
> a pretty lousey implementation...  a big step back from the C language
> standard i/o implementation of fread/fwrite that was created in the
> 1970's.

That's because it doesn't share stdio's fundamental problem, i.e. the
user-side buffering, which makes it useless for multi-user I/O. RAF
omits the user-side buffering, which you can add yourself, as you have,
and leaves the resulting multi-user problem up to you too. As shipped,
RAF works multi-user.

Re Java, the bigger mystery to me is why InputStream and OutputStream
are abstract classes instead of interfaces so that RandomAccessFile
could extend them? and/or why aren't there adapters so you could get
buffered streams out of an RAF?
Joshua Cranmer - 15 Oct 2007 01:41 GMT
> Re Java, the bigger mystery to me is why InputStream and OutputStream
> are abstract classes instead of interfaces so that RandomAccessFile
> could extend them?

My guess is so that every implementation wouldn't have to rewrite the
near-functionally equivalent overloaded read()/write() methods.

> and/or why aren't there adapters so you could get
> buffered streams out of an RAF?

I think part of this was the impetus for NIO. But I haven't really dealt
with this kind of problem, so I can't say.

Signature

Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth

Patricia Shanahan - 15 Oct 2007 02:05 GMT
>> Re Java, the bigger mystery to me is why InputStream and OutputStream
>> are abstract classes instead of interfaces so that RandomAccessFile
[quoted text clipped - 8 lines]
> I think part of this was the impetus for NIO. But I haven't really dealt
> with this kind of problem, so I can't say.

Maybe attack the problem using a byte[] to represent the record? Do a
readFully to fill a byte[] the same size as the record. Wrap the byte[]
in a byteArrayInputStream and a DataInputStream to access the fields in
the record.

Patricia


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2009 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.