Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / July 2005

Tip: Looking for answers? Try searching our database.

Fast array I/O

Thread view: 
Paul J. Lucas - 30 Jun 2005 22:53 GMT
    I want to implement a disk cache for arrays, i.e., read/write
    byte[], short[], int[], etc.  Writing byte[] is no problem, but
    all other integer arrays can't (AFAIK) be read/written as a
    single chunk.  ByteBuffer only allows one short/int to be
    read/written at a time, and that's slow.

    Is there a way to do what I want in Java or do I have to resort
    to C code via JNI?

    Notes: since this is a temporary cache, endianness is
    irrelevant.  I would like a JDK 1.4.x solution.

    - Paul
Skip - 30 Jun 2005 23:54 GMT
> I want to implement a disk cache for arrays, i.e., read/write
> byte[], short[], int[], etc.  Writing byte[] is no problem, but
> all other integer arrays can't (AFAIK) be read/written as a
> single chunk.  ByteBuffer only allows one short/int to be
> read/written at a time, and that's slow.

IntBuffer buf = byteBuffer.asIntBuffer();
buf.get(); // reads int
-> is much faster than
byteBuffer.getInt();

Besides that, IntBuffer has the bulk get/put you really want to use.

HTH
Paul J. Lucas - 01 Jul 2005 02:49 GMT
> IntBuffer buf = byteBuffer.asIntBuffer();
> buf.get(); // reads int
> -> is much faster than
> byteBuffer.getInt();

    That gets a single int, not an array as I said I wanted; that
    also doesn't actually do any file I/O as I also said I wanted.
    To be yet more specific, I need to do:

        Object readShortArray( RandomAccessFile raf, long size ) {
           byte[] bytes = new byte[ size ];
           raf.read( bytes );
           return (short[])bytes; // illegal, but what I want
        }

        void write( RandomAccessFile raf, Object obj ) {
           byte[] bytes = (byte[])obj;
           raf.write( bytes );
        }

        void callWrite() {
           short[] shorts = // ...
           write( raf, shorts );
        }

    Again, I want to read/write an entire array of int/short/long in
    a *single* I/O call.

    It would be nice if ByteBuffer had read( short[] ) but it
    doesn't.

    If this were C, it would be a no-brainer.

    - Paul
Andrey Kuznetsov - 01 Jul 2005 04:19 GMT
> That gets a single int, not an array as I said I wanted; that
> also doesn't actually do any file I/O as I also said I wanted.
[quoted text clipped - 21 lines]
> It would be nice if ByteBuffer had read( short[] ) but it
> doesn't.

see Unified I/O http://uio.imagero.com

Signature

Andrey Kuznetsov
http://uio.imagero.com Unified I/O for Java
http://reader.imagero.com Java image reader
http://jgui.imagero.com Java GUI components and utilities

Paul J. Lucas - 01 Jul 2005 06:07 GMT
So, apparently, you just copy the bytes around, e.g., for short[], you
take each short, take each byte for each short, and stuff it into a
byte array, then write the byte array.  You're still doing otherwise
unnecessary looping/copying of data, right?  I want to read/write the
raw data directly.

- Paul
Jesper Nordenberg - 01 Jul 2005 11:58 GMT
> So, apparently, you just copy the bytes around, e.g., for short[], you
> take each short, take each byte for each short, and stuff it into a
[quoted text clipped - 3 lines]
>
> - Paul

See http://java.sun.com/j2se/1.5.0/docs/api/java/nio/ShortBuffer.html#get(short[])

/JN
Paul J. Lucas - 01 Jul 2005 20:44 GMT
> See http://java.sun.com/j2se/1.5.0/docs/api/java/nio/ShortBuffer.html#get(short[])

    I said *fast* I/O.  If you actually look at the source code, it
    loops and does one short at a time.

    - Paul
Mark Thornton - 01 Jul 2005 23:08 GMT
>>See http://java.sun.com/j2se/1.5.0/docs/api/java/nio/ShortBuffer.html#get(short[])
>
>     I said *fast* I/O.  If you actually look at the source code, it
>     loops and does one short at a time.
>
>     - Paul

You should ignore the Java source code in this case. When the byte
ordering of the underlying ByteBuffer matches the native byte ordering,
the JIT seems to replace the code with something much faster. I suggest
you try it out. When you get it right the speed magically jumps.

Mark Thornton
Paul J. Lucas - 02 Jul 2005 18:09 GMT
> >       I said *fast* I/O.  If you actually look at the source code, it
> >       loops and does one short at a time.
>
> You should ignore the Java source code in this case. When the byte ordering
> of the underlying ByteBuffer matches the native byte ordering, the JIT seems
> to replace the code with something much faster.

    That's just it: I want it fast on all Macs and PCs (Intel Macs
    won't be around in significant numbers for several years).
    Endianness for my application is irrelevant.

    - Paul
Andrey Kuznetsov - 01 Jul 2005 20:52 GMT
> See
> http://java.sun.com/j2se/1.5.0/docs/api/java/nio/ShortBuffer.html#get(short[])

nio doesnot read short array direct, as Paul it want.

If you look at source then you will that ShortBuffer.get(short[])
calls in loop ShortBuffer.get() which is abstract.
One of implementations you can find in DirectShortBufferS.
it calls native method getShort().
So every call to ShortBuffer.get(short[]) results in multiple native calls.
Which can be expensive.

If you see at source of Unified I/O you will see that uio uses byte array to
buffer data.
This byte array is filled with call to RandomAccessFile.read(byte [], int,
int) which calls
native method RandomAccessFile.readBytes(byte [], int, int).

The latter method looks like native bulk read, which is very fast.

Signature

http://uio.imagero.com Unified I/O for Java
http://reader.imagero.com Java image reader
http://jgui.imagero.com Java GUI components and utilities
new - http://forum.imagero.com

Paul J. Lucas - 01 Jul 2005 22:25 GMT
> The latter method looks like native bulk read, which is very fast.

    But you still loop to reconstruct shorts from bytes when you
    read the byte[].

    Ideally, I want *no* loops.

    - Paul
Andrey Kuznetsov - 01 Jul 2005 20:32 GMT
> So, apparently, you just copy the bytes around, e.g., for short[], you
> take each short, take each byte for each short, and stuff it into a
> byte array, then write the byte array.  You're still doing otherwise
> unnecessary looping/copying of data, right?  I want to read/write the
> raw data directly.

Paul, the impact on copying is very little and win from buffering is very
big.
http://uio.imagero.com/performance.html

Signature

Andrey Kuznetsov
http://uio.imagero.com Unified I/O for Java
http://reader.imagero.com Java image reader
http://jgui.imagero.com Java GUI components and utilities

Esmond Pitt - 01 Jul 2005 08:22 GMT
    MappedByteBuffer    mbb = fileChannel.map(...);
    IntBuffer    ib = mbb.asIntBuffer();
    int[] ia = new int[...];
    int count = ib.get(ia);

(modulo a few flips). There is only one I/O operation, at the 'get'; the
the rest is just mapping.
Paul J. Lucas - 01 Jul 2005 17:23 GMT
>         MappedByteBuffer        mbb = fileChannel.map(...);
>         IntBuffer       ib = mbb.asIntBuffer();
[quoted text clipped - 3 lines]
> (modulo a few flips). There is only one I/O operation, at the 'get'; the
> the rest is just mapping.

    The problem with using a MappedByteBuffer is that the bytes
    remain in memory until the GC decides to flush them out to disk.
    This takes up memory.  My original goal (which I described) is
    to write a disk cache.  That means that I want to force an int[]
    out to disk *now*.  Once that happens, the int[] is released.
    Even if I were to use the force() method, there's still an
    in-memory mapping reserved for the int[].  That defeats the
    purpose of having a disk cache.  The int[] either to be in
    memory *or* on disk, but never both.

    - Paul
Skip - 01 Jul 2005 17:48 GMT
> The int[] either to be in memory *or* on disk, but never both.

This is Java, you never know how long something resides in memory.

You might try System.gc() to give a subtile hint to the VM.
Skip - 01 Jul 2005 10:20 GMT
> > IntBuffer buf = byteBuffer.asIntBuffer();
> > buf.get(); // reads int
> > -> is much faster than
> > byteBuffer.getInt();
>
> That gets a single int, not an array as I said I wanted

Yeah, you snipped my last line:

>> Besides that, IntBuffer has the bulk get/put you really want to use.

Read.
Paul J. Lucas - 01 Jul 2005 17:24 GMT
> > > IntBuffer buf = byteBuffer.asIntBuffer();
> > > buf.get(); // reads int
[quoted text clipped - 8 lines]
>
> Read.

    I don't believe MappedByteBuffers will work.  See my other
    follow-up as to why not.

    - Paul
Thomas Weidenfeller - 01 Jul 2005 08:20 GMT
>     I want to implement a disk cache for arrays, i.e., read/write
>     byte[], short[], int[], etc.  Writing byte[] is no problem, but
>     all other integer arrays can't (AFAIK) be read/written as a
>     single chunk.  ByteBuffer only allows one short/int to be
>     read/written at a time, and that's slow.

    ByteBuffer.asShortBuffer()
    ByteBuffer.asIntBuffer()
    ByteBuffer.asLongBuffer()
    ByteBuffer.asFloatBuffer()
    ByteBuffer.asDoubleBuffer()
    ByteBuffer.asCharBuffer()

Please read the API documentation for details.

>     Is there a way to do what I want in Java or do I have to resort
>     to C code via JNI?

Implementations of ByteBuffer deep down use JNI. But, they also do copy
data around to a certain extend, e.g. when they deem it necessary to
change the endiness of data for whatever reason (I never took the time
to figure out when they do this).

>     Notes: since this is a temporary cache, endianness is
>     irrelevant.  I would like a JDK 1.4.x solution.

Use memory-mapped-io with a MemoryByteBuffer, and ShortBuffers,
IntBuffers, LongBuffers, etc. as views on this one.

/Thomas

Signature

The comp.lang.java.gui FAQ:
ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq
http://www.uni-giessen.de/faq/archiv/computer-lang.java.gui.faq/

Paul J. Lucas - 01 Jul 2005 17:36 GMT
> Please read the API documentation for details.

    I did prior to posting.

> Use memory-mapped-io with a MemoryByteBuffer, and ShortBuffers,
> IntBuffers, LongBuffers, etc. as views on this one.

    I don't believe ByteBuffers will work.  A MappedByteBuffer, or
    more specifically mmap(2), is designed to be able to read large
    files, possible randomly, on an as-needed based, i.e., you fetch
    a byte: if it's not in memory, a page-fault occurs, and it's
    swapped in.  The file is transparently treated as though it *is*
    memory.

    For writing a disk cache, I don't think the mmap(2) model works
    too well.  When I decide I want to write an array to disk, I
    want it written immediately.  Once written, its in-memory
    counterpart is freed immediately.

    Using mmap(2), and therefore MappedByteBuffer, I'm at the mercy
    of the Java garbage collector to decide what and when to release
    from memory.  If memory gets tight (which is why I want a disk
    cache in the first place), there's no guarantee that the GC will
    free what I want it to free.  It might very well decide to free
    other, unrelated stuff in memory, e.g., other data structures
    using WeakReferences, and not the arrays I want it to free.

    - Paul
Gitta Zahn - 01 Jul 2005 21:31 GMT
>     I don't believe ByteBuffers will work.  A MappedByteBuffer, or
>     more specifically mmap(2), is designed to be able to read large
[quoted text clipped - 7 lines]
>     want it written immediately.  Once written, its in-memory
>     counterpart is freed immediately.

The operating system will ensure the changed pages are written back to
disk. On Solaris fsflush cyclical scans the pages and writes out
dirty pages to disk. You can enforce a previous change by calling
the MappedByteBuffer.force() operation as you already mentioned.

>     Using mmap(2), and therefore MappedByteBuffer, I'm at the mercy
>     of the Java garbage collector to decide what and when to release
[quoted text clipped - 3 lines]
>     other, unrelated stuff in memory, e.g., other data structures
>     using WeakReferences, and not the arrays I want it to free.

Which memory - the memory used for Java heap, the native memory of the
process, the overall memory on the machine ?
The overall memory on the machine is managed by the operating system,
page are swapped out on demand.
The buffer for such a MappedByteBuffer should reside outside the normal
Java heap. So a delayed GC of such MappedByteBuffers leads to a
situation, where the native memory is not unmapped fast enough.
This would be a problem if your process runs out of the possible
virtual address space. In this case you could try the 64 bit Java.

The other solution would be to do this directly with JNI operations
which call the mmap() and munmap() and provide the file as
java.nio.ByteBuffer via NewDirectByteBuffer() to the Java side.
But I think here an operation is missing which tells the JVM that
the ByteBuffer is now invalidated,

Gitta
Thomas Weidenfeller - 04 Jul 2005 10:53 GMT
>>Please read the API documentation for details.
>
>     I did prior to posting.

So why did you claim you can't write short or int arrays via a ByteBuffer?

> A MappedByteBuffer, or
>     more specifically mmap(2), is designed to be able to read large
>     files,

It is suitable for reading and writing. Why do you cite a man page you
apparently have not read?

>     For writing a disk cache, I don't think the mmap(2) model works
>     too well.  When I decide I want to write an array to disk, I
>     want it written immediately.

MappedByteBuffer.force()

So much for you reading the API documentation prior to posting.

/Thomas

Signature

The comp.lang.java.gui FAQ:
ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq
http://www.uni-giessen.de/faq/archiv/computer-lang.java.gui.faq/

Paul J. Lucas - 04 Jul 2005 15:57 GMT
> >>Please read the API documentation for details.
> >
> >       I did prior to posting.
>
> So why did you claim you can't write short or int arrays via a ByteBuffer?

    Which part of "Fast" in my subject of "Fast array I/O" is
    unclear?  ByteBuffer's implementation loops and does endian
    conversions.

> > A MappedByteBuffer, or
> >       more specifically mmap(2), is designed to be able to read large
> >       files,
>
> It is suitable for reading and writing. Why do you cite a man page you
> apparently have not read?

    I've read it many times.  It's not particularly well-suited for
    disk caches.

> >       For writing a disk cache, I don't think the mmap(2) model works
> >       too well.  When I decide I want to write an array to disk, I
> >       want it written immediately.
>
> MappedByteBuffer.force()

    The problem with force() is that it doesn't free the in-memory
    copy.

    - Paul
Mark Thornton - 04 Jul 2005 19:53 GMT
>>>>Please read the API documentation for details.
>>>
[quoted text clipped - 5 lines]
>     unclear?  ByteBuffer's implementation loops and does endian
>     conversions.

As I keep telling you, it ONLY does that where endian conversions are
necessary. The short/int arrays are of necessity in native ordering, if
the ByteBuffer is also of native ordering then it uses (native) bulk
copying methods. What appears in the Java library source code is
irrelevant here.

Mark Thornton
Thomas Weidenfeller - 05 Jul 2005 07:58 GMT
>     Which part of "Fast" in my subject of "Fast array I/O" is
>     unclear?  ByteBuffer's implementation loops and does endian
>     conversions.

Which part of "You are an idiot" don't you understand? A five second dig
into the implementation of the S-U-B-C-L-A-S-S-E-S like MappedByteBuffer
will show you that they use native code.

But I guess you have zero interest in getting your problem fixed.

/Thomas

Signature

The comp.lang.java.gui FAQ:
ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq
http://www.uni-giessen.de/faq/archiv/computer-lang.java.gui.faq/

Roedy Green - 01 Jul 2005 19:51 GMT
On Thu, 30 Jun 2005 21:53:00 GMT,
pauljlucas.removethis@removethistoo.mac.com (Paul J. Lucas) wrote or
quoted :

>    I want to implement a disk cache for arrays, i.e., read/write
>    byte[], short[], int[], etc.  Writing byte[] is no problem, but
[quoted text clipped - 4 lines]
>    Is there a way to do what I want in Java or do I have to resort
>    to C code via JNI?

see http://mindprod.com/applets.html#fileio.html

it will generate code for the various techniques.  I suggest
serialising the array. IT goes out in one big chunk of io.
Then there are the nio methods that read without converting endianness
until you actually use the fields, good for skipping over large junks.

Signature

Bush crime family lost/embezzled $3 trillion from Pentagon.
Complicit Bush-friendly media keeps mum. Rumsfeld confesses on video.
http://www.infowars.com/articles/us/mckinney_grills_rumsfeld.htm

Canadian Mind Products, Roedy Green.
See http://mindprod.com/iraq.html photos of Bush's war crimes

Paul J. Lucas - 01 Jul 2005 20:43 GMT
> see http://mindprod.com/applets.html#fileio.html

    The requested URL /applets.html was not found on this server.

    - Paul
Knute Johnson - 01 Jul 2005 22:55 GMT
What about ObjectOutputStream?

Signature

Knute Johnson
email s/nospam/knute/

Paul J. Lucas - 02 Jul 2005 18:09 GMT
> What about ObjectOutputStream?

    I need I/O -- note the 'I' part too.

    - Paul
Joan - 03 Jul 2005 02:39 GMT
> > What about ObjectOutputStream?
>
> I need I/O -- note the 'I' part too.
>
> - Paul

How about a pipe?
Paul J. Lucas - 03 Jul 2005 17:29 GMT
> > > What about ObjectOutputStream?
> >
> > I need I/O -- note the 'I' part too.

> How about a pipe?

    A Pipe is  unidirectional.  I/O is bidirectional.  Yet again: I
    want fast (no loops, no endian conversions) Object I/O (that's
    input *and* output*) to a *single* file (like RandomAccessFile).

    Are we clear yet?

    - Paul
Mark Thornton - 03 Jul 2005 19:52 GMT
>>>>What about ObjectOutputStream?
>>>
[quoted text clipped - 9 lines]
>
>     - Paul

Then what you want is NIO via ByteBuffers with native byte ordering.

Mark Thornton
Paul J. Lucas - 04 Jul 2005 15:58 GMT
> Then what you want is NIO via ByteBuffers with native byte ordering.

    Read the Java source code for ByteBuffers.  They still loop.

    - Paul
Mark Thornton - 04 Jul 2005 19:56 GMT
>>Then what you want is NIO via ByteBuffers with native byte ordering.
>
>     Read the Java source code for ByteBuffers.  They still loop.
>
>     - Paul

As I keep saying, at run time the JVM implements that with native bulk
copies so long as native byte ordering is used. Forget what appears in
the source code.

Mark Thornton
Knute Johnson - 03 Jul 2005 16:10 GMT
>>What about ObjectOutputStream?
>
>     I need I/O -- note the 'I' part too.
>
>     - Paul

I don't know if you are just being a jerk or are stupid.

Look at ObjectInputStream too!

Signature

Knute Johnson
email s/nospam/knute/

Paul J. Lucas - 03 Jul 2005 17:26 GMT
> >>What about ObjectOutputStream?
> >
> >       I need I/O -- note the 'I' part too.
> >
> I don't know if you are just being a jerk or are stupid.

    I don't know the same about you.  I've *clearly* outlined what I
    want.

> Look at ObjectInputStream too!

    The problem with Java's class hierarchy is that I want a
    *single* file-handle to allow read *and* write access.  It's a
    disk cache -- you need to write sometimes, and read back
    sometimes, hence my current use of RandomAccessFile.

    - Paul
Skip - 03 Jul 2005 19:16 GMT
> > >>What about ObjectOutputStream?
> > >
[quoted text clipped - 4 lines]
> I don't know the same about you.  I've *clearly* outlined what I
> want.

So... ? It's not wrong to also use your own brains, instead of expecting
100% accurate answers.

> > Look at ObjectInputStream too!
>
> The problem with Java's class hierarchy is that I want a
> *single* file-handle to allow read *and* write access.  It's a
> disk cache -- you need to write sometimes, and read back
> sometimes, hence my current use of RandomAccessFile.

If you'd have combined a few answers here, you'd have known the answer.

Write a bridge between java.io Streams and RandomAccessFile. Once you're
there, you can open and close streams at any point of the file.

And now please stop being so damn arrogant, because it stinks.

HTH
Knute Johnson - 04 Jul 2005 01:21 GMT
You wrote you wanted to read/write complete arrays to temporary storage
as fast as possible.  The ObjectOutputStream and ObjectInputStream will
do that with a minimum of code and plenty of speed.  You didn't specify
how large your arrays are but a million elements can be read or written
in a couple of seconds.  If you need multiple records, just use multiple
files.  If your arrays are much smaller, keep several in a file and
read/write all of them.  The performance hit will be minimal.  No JNI,
no C, no complicated messing around just simple Java.

Signature

Knute Johnson
email s/nospam/knute/

Paul J. Lucas - 04 Jul 2005 16:04 GMT
> You wrote you wanted to read/write complete arrays to temporary storage
> as fast as possible.  The ObjectOutputStream and ObjectInputStream will
> do that with a minimum of code and plenty of speed.

    Can you use an ObjectOutputStream and an ObjectInputStream on
    the same physical file concurrently?  That's what I'd need.  (It
    seems wasteful to have to separate objects.)

    - Paul
Paul J. Lucas - 04 Jul 2005 16:08 GMT
> > You wrote you wanted to read/write complete arrays to temporary storage
> > as fast as possible.  The ObjectOutputStream and ObjectInputStream will
[quoted text clipped - 3 lines]
>         the same physical file concurrently?  That's what I'd need.  (It
>         seems wasteful to have to separate objects.)

    Actually, ObjectOutputStream and ObjectInputStream use the Java
    serialization stuff.  That means that endian conversions are
    done since the whole serialization framework enables you to
    serialize objects on one platform and deserialize them on
    another, right?  That's not what I want.

    - Paul
Knute Johnson - 04 Jul 2005 16:43 GMT
>>>You wrote you wanted to read/write complete arrays to temporary storage
>>>as fast as possible.  The ObjectOutputStream and ObjectInputStream will
[quoted text clipped - 11 lines]
>
>     - Paul

I don't know if that is true or not and I'm not going into the source to
find out.

If you are only writing for one platform, why don't you just write it in
assembler.  That will be much faster and the rest of us won't have to
try to help you any more.

Signature

Knute Johnson
email s/nospam/knute/

Knute Johnson - 04 Jul 2005 19:02 GMT
>>>> You wrote you wanted to read/write complete arrays to temporary
>>>> storage as fast as possible.  The ObjectOutputStream and
[quoted text clipped - 19 lines]
> assembler.  That will be much faster and the rest of us won't have to
> try to help you any more.

I thought I would look at a file created with ObjectOutputStream and the
byte order is BigEndian on my Intel Windows machine.  That is not native.

Signature

Knute Johnson
email s/nospam/knute/

Paul J. Lucas - 06 Jul 2005 11:44 GMT
> >>        Can you use an ObjectOutputStream and an ObjectInputStream on
> >>        the same physical file concurrently?  That's what I'd need.  (It
[quoted text clipped - 8 lines]
> I don't know if that is true or not and I'm not going into the source to
> find out.

    I'm pretty sure it's true.

> If you are only writing for one platform, why don't you just write it in
> assembler.

    Because I'm not writing it for one platform.  However, on a
    given run, only the current platform is relevant; hence endian
    conversions are unnecessary.

    - Paul
Knute Johnson - 06 Jul 2005 17:11 GMT
>>>>       Can you use an ObjectOutputStream and an ObjectInputStream on
>>>>       the same physical file concurrently?  That's what I'd need.  (It
[quoted text clipped - 19 lines]
>
>     - Paul

Actually I looked at the docs and ObjectOutputStream uses
DataOutputStream for the conversions and as a result is always BigEndian.

Signature

Knute Johnson
email s/nospam/knute/

William Brogden - 04 Jul 2005 16:14 GMT
>> You wrote you wanted to read/write complete arrays to temporary storage
>> as fast as possible.  The ObjectOutputStream and ObjectInputStream will
[quoted text clipped - 5 lines]
>
>     - Paul

I am coming late to this discussion so perhaps this has already come up.

Have you created a test frame to do actual measurements on a real system
with realistic data, or has all this been pure speculation?

Bill
Andrew Thompson - 05 Jul 2005 11:10 GMT
> Have you created a test frame to do actual measurements on a real system
> with realistic data, or has all this been pure speculation?

The way I read this conversation (I have been following it
with a morbid fascination) 'b'..

Signature

Andrew Thompson
http://www.PhySci.org/codes/  Web & IT Help
http://www.PhySci.org/  Open-source software suite
http://www.1point1C.org/  Science & Technology
http://www.LensEscapes.com/  Images that escape the mundane

Knute Johnson - 04 Jul 2005 16:37 GMT
>>You wrote you wanted to read/write complete arrays to temporary storage
>>as fast as possible.  The ObjectOutputStream and ObjectInputStream will
[quoted text clipped - 5 lines]
>
>     - Paul

That's ridiculous.  If you are still using the array, don't store it.
Two objects is not wasteful, you'll create a lot more than two in any
program you ever write.

Signature

Knute Johnson
email s/nospam/knute/

Joan - 04 Jul 2005 19:56 GMT
> That's ridiculous.  If you are still using the array, don't store it.
> Two objects is not wasteful, you'll create a lot more than two in any
> program you ever write.

How come so much discussion on implementation and nothing on requirements.
Like HOW fast??? It is like saying I need to get to Detroit, but not telling
how fast,
or when you need to get there, but bitching about what sort of tires the bus
has.
Skip - 04 Jul 2005 21:20 GMT
> > That's ridiculous.  If you are still using the array, don't store it.
> > Two objects is not wasteful, you'll create a lot more than two in any
[quoted text clipped - 5 lines]
> or when you need to get there, but bitching about what sort of tires the bus
> has.

Yes, and the OP obviously doesn't ahve a clue what the true bottleneck is:
the harddisk. You might speedup serialisation some way by factor 10, from
1.0ms to 0.1ms, but does it matter when writing it to disk takes 25ms?

And the demanding attitude is really silly. As he's probably payed to get
that code working, he shouldn't expect us to line up with all answers.

And still we all line up, with answers :o)
Paul J. Lucas - 06 Jul 2005 11:43 GMT
> How come so much discussion on implementation and nothing on requirements.
> Like HOW fast?

    Ideally, as fast as:

        // Good ol' C code
        void writeShort( int fd, short a[], int len ) {
           write( fd, a, len * sizeof(short) );
        }

        void readShort( int fd, short a[], int len ) {
           read( fd, a, len * sizeof(short) );
        }

    It's pretty hard to get faster than that.

    - Paul
Andrey Kuznetsov - 07 Jul 2005 23:27 GMT
>> How come so much discussion on implementation and nothing on
>> requirements.
[quoted text clipped - 12 lines]
>
> It's pretty hard to get faster than that.

then you should use C or JNI .
However before you do it you should read this:
http://java.sun.com/docs/books/performance/1st_edition/html/JPIOPerformance.fm.html

Signature

Andrey Kuznetsov
http://uio.imagero.com Unified I/O for Java
http://reader.imagero.com Java image reader
http://jgui.imagero.com Java GUI components and utilities

Joan - 08 Jul 2005 21:12 GMT
> >> How come so much discussion on implementation and nothing on
> >> requirements.
[quoted text clipped - 12 lines]
> >
> > It's pretty hard to get faster than that.

I just noticed that I don't see any reference to stuff like "open"
so maybe your so called high speed "file" is being accessed
via a 110 baud modem.
Andrew Thompson - 08 Jul 2005 22:38 GMT
Note that..

..made a post that stated..

> then you should use C or JNI .
> However before you do it you should read this:
> http://java.sun.com/docs/books/performance/1st_edition/html/JPIOPerformance.fm.html

..as opposed to, ..*any* of the words attributed to Andrey!

Signature

Andrew Thompson
http://www.PhySci.org/codes/  Web & IT Help
http://www.PhySci.org/  Open-source software suite
http://www.1point1C.org/  Science & Technology
http://www.LensEscapes.com/  Images that escape the mundane



Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2009 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.