The most efficient usage of your time is not to reinvent wheels.

Having multiple threads doing I/O to the same disk is likely to slow
things down.

I see no benefit in reading a whole file into memory.

You don't need the JVM to be doing conversion to UTC-16, or pointless
line-oriented processing (e.g. scanning for line-endings).

cat  sourceFileOne.txt sourceFileTwo.txt ... targetFile.txt

or

copy sourceFileOne.txt+sourceFileTwo.txt ... targetFile.txt

depending on operating system

I wouldn't reinvent this wheel but if you are doing it I suggest you
treat the files as binary not as text (especially not using anything
that translates encodings). Reading in large fixed-size chunks would
seem to be sensible. Given that the task is I/O bound I wouldn't try too
hard to optimise anything else.

What encoding are your text files in? If the source and target files are  
in the same encoding, and do not have a BOM character at the beginning of  
the file, then a binary transfer is the way to go. Take a look at  
java.nio.channels.FileChannel.transferTo / transferFrom
http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html#transfe
rTo(long,  
long, java.nio.channels.WritableByteChannel)

As those methods should give you very fast file content transferal for  
binary data.

If you need to deal with different encodings (from your example usage, you  
might check to see if your source files were using different BOMs), then  
reading a block of characters (decoding from source), and writing them  
back to the target (encoding them with the target file's encoding) may be  
more appropriate. If they all have the same encoding, but use BOMs, then  
you can use a binary transfer, skipping the BOM character from all but the  
first source file.

Reading & decoding blocks of data will also give you the flexiblity to  
support more options, such as reading seperately gzip'ed log files, and  
writing them out as a single gzip'ed text file.

HTH,

-Zig

    The fastest and most efficient way of all is -- Don't Do That.
Do you really *need* a second copy of the contents of all those
files?  Or could you use a java.io.SequenceInputStream to read
the originals /in situ/?

    If you actually do need to concatenate, it's highly unlikely
that anything you can do in Java will be as fast as the platform's
own file-concatenation utility.  Those beasts tend to be heavily
optimized, using platform-specific trickery and undocumented API's
to move the data from hither to yon at great speed.  If it's speed
you care about, spend your time figuring out how to launch the
native utility instead of spending it trying to optimize an
alternative that's hobbled by portability concerns.

Why not use concat task that comes with ant? Or if you can use shell on
a nix box, use "cat". Or install cat binary from cygwin on the windows
box (the list goes on). There are many solutions out there, the least
recommended being writing something like this from scratch (unless you
are doing this just for learning or for fun).
Abhijat

On Wed, 2 Jul 2008 09:51:55 -0700 (PDT), sasuke
<database666@gmail.com> wrote, quoted or indirectly quoted someone who
said :

1. If you want a platform-specific solution, you could spawn a command
processor shell.

2. The simplest code would just be to read each file with a
BufferedReader using a whacking huge buffersize and write in turn to a
bufered  output. see http://mindprod.com/applet/fileio.html for
sameple code. That has needless overhead for converting from bytes to
char and back, though it theory you could concatenate files of
different encodings if you knew what they were.

3.  if you read the files as raw bytes rather than chars, you know
their precise lengths, and the offset where they will fit in the final
file.  You could use random access to implement your thread idea.
However, I doubt the game will be worth the candle unless the files to
be gathered  live on different _physical_ drives. All you will succeed
in doing is jerking the heads all over.

4. If you want a canned solution, use the FileTransfer class
downloadable from
http://mindprod.com/products.html#FILETRANSFER

It does it rapidly in large raw-byte chunks.

// test FileTransfer.append
import com.mindprod.filetransfer.FileTransfer;
import java.io.File;
public class Concat
  {
  /**
   * test harness to concatenate c onto the end of b, leaving the
result in a.
   *
   * @param args not used
   */
  public static void main ( String[] args )
     {
     File a = new File ("C:/temp/temp.txt");  // does not exist yet
     File b = new File ("E:/mindprod/feedback/peaceincorrect.html");
     File c = new File ("E:/mindprod/jgloss/j.html");

     FileTransfer ft = new FileTransfer ( 50000 /* buffsize */ );
     // source, target
     ft.append( b, a );
     ft.append( c, a );
     }
  }