at-first, you don't need such huge buffers - they can make you slow.
at-second, you can try to use Unified I/O - it makes all buffering for you.
In some cases Unified I/O may give you speedup up to 120 times.
If you use readLine, you become String, this can make it slow.
With Unified I/O you can read line to byte array.
Its free and open source.

In 25 years of working with computers professionally, a good bit of it as an
instructor of computer courses, I've never seen a requirement to do anything
like what you are talking about, namely slicing a file vertically. I suppose
the main intent is just to make you learn to handle memory and file I/O but
you'd think the instructor of your course could come up with a more
realistic problem that would help you develop those same skills....

---
Rhino

I suspect that, as Andrey has already said, your buffers are oversized.  If you
arrived at those sizes by experiment then kudos to you, and please ignore the
rest of the paragraph.  If not then I'd try using 1M for all of them, or even
half that.  I've never seen a significant performance gain by using buffers
over a M (except when writing to a tape drive).

One thing to consider is that as you write the 200 output files, you will be
making the disk heads move a /lot/ and not sequentially, and that may well be
slowing your program down substantially -- or there again, it may not.  I'd try
doing this in N passes, each extracting 200/N columns.  I don't know what the
best value of N is, it may be 1 as you are currently assuming ;-) but I'd try
increasing it to -- say -- 20 and see what happens.  It's even possible that
the optimum will be 200.  Considering disk movements can be a big win -- I once
rewrote a single pass program that would have taken >10 hours to run, into a
multi-pass (near-)equivalent that ran in about 3 minutes...

A more sophisticated multi-pass version would be to read in the input file in
giant chunks (100 M, say), and then do multiple passes over that, appending the
data from that chunk to each output file in turn.  You might find it worthwhile
splitting up the data in the chunk at the beginning rather than duplicating
that effort on each pass, but that would massively increase your object counts
and thus reduce the size of the chunk that you could use.  A tradeoff and only
experiment (or careful /quantative/ analysis) can tell you which is better.

You may also be able to micro-optimise the splitting of the file into lines and
sub-strings, but I find it hard to believe that that would have any significant
effect in comparison with IO costs.

   -- chris

"Kevin" <kaidizhao@yahoo.com.sg> wrote or quoted in
Message-ID: <1139009028.703566.52430@z14g2000cwz.googlegroups.com>:

I assume the followings:

a) Space is valid character.
b) Line divider is "\r\n" or '\n'.

Maybe,  it is as like:

<code>
(this codes is not tested).

public class MainFrame {
                                     
    static InputStream in;
    static final int colNum = 200;
    static OutputStream[] out = new OutputStream[colNum];
   
    public static void main(String[] args) {
       
        try {
            initializeStream();
           
            byte[] b = new byte[1];
            ByteBuffer buf = new ByteBuffer(512);
            int colIndex = 0;
           
            while (true) {
               
                if (in.read() < 0) {
                   
                    break;
                }
                switch (b[0]) {
                case ',':
                    out[colIndex++].write(buf.get(), 0, buf.size());
                    buf.clear();
                    break;

                case '\r': // skip
                    break;
                   
                case '\n':
                    colIndex = 0;
                    break;
                   
                default:
                    buf.add(b[0]);
                    break;
                }
            }
        }
        catch (Exception e) {
            // process exception.
        }
        finally {
            closeStream();
        }
    }
   
    static void initializeStream() throws Exception {
       
        in = getInputStream();
        Arrays.fill(out, null);
       
        for (int i = 0; i < out.length; i++) {
           
            out[i] = getOutputStream(i);
        }
    }

    static void closeStream() {
       
        try {
            if (in != null) in.close();
           
            for (int i = 0; i < out.length; i++) {
               
                if (out[i] != null) out[i].close();
            }
        }
        catch (Exception e) {
        }
    }
   
    static InputStream getInputStream() throws Exception {

        File f = new File("your input file");
        return new BufferedInputStream(new FileInputStream(f));
    }
   
    static OutputStream getOutputStream(int index) throws Exception {
       
        File f = new File("your output file" + index + ".txt");
        return new BufferedOutputStream(new FileOutputStream(f));
    }
   
    static class ByteBuffer {
       
        int pos;
        byte[] buf;
       
        public ByteBuffer(int initialCapacity) {
           
            buf = new byte[initialCapacity];
        }
       
        void add(byte b) {
           
            if (pos >= buf.length) {
               
                resize();
            }
            buf[pos++] = b;
        }
       
        void resize() {
           
            byte[] b = new byte[buf.length * 2];
            System.arraycopy(buf, 0, b, 0, buf.length);
           
            buf = b;
        }
       
        byte[] get() {
           
            return buf;
        }
       
        int size() {
           
            return pos;
        }
       
        void clear() {
           
            pos = 0;
        }
    }
}

</code>