how to improve transfering file's efficiency

Thread view:

JTL.zheng - 19 Nov 2006 10:23 GMT

I have write program for client to client file transmission
but it seems unefficient ,the transmission speed is low
and it take too much CPU resource

here are the codes(sender's and receiver's)
are there some ways to improve it about the transfer speed and cpu?

sender's code:
-------------------------------------------------------
input_file = new BufferedInputStream(new FileInputStream(file));
output_file = new
BufferedOutputStream(socket_file.getOutputStream());

byte[] buffer = new byte[10240];
int readed;
while ( (readed = input_file.read(buffer)) != -1) {
output_file.write(buffer, 0, readed);
}

output_file.flush();
-------------------------------------------------------

receiver's code:
-------------------------------------------------------
output_file = new BufferedOutputStream(new
FileOutputStream(file));
input_file = new
BufferedInputStream(socket_file.getInputStream());

byte[] buffer = new byte[10240];
int readed;
while ( (readed = input_file.read(buffer)) != -1) {
output_file.write(buffer, 0, readed);
}

output_file.flush();
-------------------------------------------------------

Thank you very much in advance

Reply to this Message

ChrisWSU - 19 Nov 2006 16:00 GMT

this is over networking? NIO could potentially increase speed, using
directly allocated buffers and such.

Id suggest decreasing the size of the buffer, reading it all at once
and sending all at once wont necessarily be more efficient because some
of the data could be being transfered on the wire while your doing the
reads from disk. generally the issue would be more about your network
speeds then the java code.

also you could try wraping it in a gzip stream to decrease the amount
being sent over the bottleneck (the network), although that increase
the load on the CPU.

Reply to this Message

JTL.zheng - 21 Nov 2006 04:04 GMT

Thank you very much for your help

yes,it's over networking.
I have test to decrease the buffer
10240 and 1024 is almost the same speed
but if decrease to 10, the speed will slow to 2/3

the file transfer speed is much lower than my network(LAN)
so I think maybe it could get a higher speed

>also you could try wraping it in a gzip stream to decrease the amount

what's a "gzip stream"?

Reply to this Message

lecointe.pascal@wanadoo.fr - 21 Nov 2006 07:00 GMT

JTL.zheng a ?crit :

> Thank you very much for your help
>
[quoted text clipped - 9 lines]
>
> what's a "gzip stream"?

If you use a byte array to read (or write) in your stream, you don't
need to use a BufferedStream.

A simple stream is better, because in this case a BufferedStream must
copy your buffer in his self buffer, a not necessary operation

Reply to this Message

Chris Uppal - 21 Nov 2006 10:14 GMT

> the file transfer speed is much lower than my network(LAN)
> so I think maybe it could get a higher speed

How fast is your LAN ? How fast is your code running ?

Also, do you have any other software which can transfer data between those two
machines over the same LAN at a noticeably faster rate ? It may be a problem
with the LAN itself rather than either your code or the Java platform.

I can't see anything much wrong with the code, so I'd expect it to pretty much
saturate either the LAN itself or the network cards on one or the other
machine. I would /not/ expect it to max out the CPU.

It might be worth getting rid of the buffering around the InputStream. I don't
think that'll make much difference, but it may be worth the trial. Since you
are using a large read buffer anyway, the extra buffering will only incur extra
copying costs. I would leave the buffers on the output side in place, so that
if you do see short reads (just a few bytes at a time), they won't translate
into short writes too.

Another thing that might be worth trying is to remove the write()s in the
sender, and see how fast that can run when all it's doing is reading the file.
Similarly, remove the writes() in the reciever and see how fast that can run
when it is only reading from the LAN and not writing a file too.

(I would also, by now, have fired up Ethereal[*] to see if there was anything
odd happening at the network level -- very small packets, or lots of retries,
or something of that kind.)

-- chris

[*] Apparently now also known as "Wireshark" -- silly name.

Reply to this Message

JTL.zheng - 03 Dec 2006 07:54 GMT

Thank you very much for yours advices

I have test several modes(have/not have I/O Buffere) and the size of
the buffer
and get rid of the writing file in recever when test
now the speed is avg. 10.2MB/s (my LAN is 100Mbps)

my new codes is:
--------------------------------
input_file = new FileInputStream(file);
output_file = socket_file.getOutputStream();

byte[] buffer = new byte[2048000];
int readed;
while ( (readed = input_file.read(buffer)) != -1) {
output_file.write(buffer, 0, readed);
}
--------------------------------
I get rid of the BufferedInput/BufferedOutput as it doesn't improve the
transfer speed
and take more CPU time.

I also change the size of the buffer
when 1KB,the speed is 5.4MB/s
10KB is 7.7 MB/s
but after 100KB(500KB,1MB,2MB,5MB,10MB)
the speed is standing in 10.2MB/s

and it take less cpu when the buffer is bigger
after 1MB it stand in 80%

so my default buffer is 2048000B
(I also add a JComboBox using to set the buffer size)

but the cpu is still a litte high
it is avg. 78% (my cpu is AMD Athlon 2500+)
is that any way to low down the cpu?

Reply to this Message

Chris Uppal - 04 Dec 2006 16:40 GMT

> I have test several modes(have/not have I/O Buffere) and the size of
> the buffer
> and get rid of the writing file in recever when test
> now the speed is avg. 10.2MB/s (my LAN is 100Mbps)

That's reasonable (but see below). Once you allow for the overheads of TCP,
and IP, you are not going to get very much closer to the theoretical bandwidth
of 12.5 MB/s (or 11.9 if you use binary megabytes).

> I also change the size of the buffer
> when 1KB,the speed is 5.4MB/s
> 10KB is 7.7 MB/s
> but after 100KB(500KB,1MB,2MB,5MB,10MB)
> the speed is standing in 10.2MB/s

That's a bit odd. In a test on my LAN I don't have to use nearly that big a
buffer to get reasonable speeds. Using a 1024 byte[] read/write buffer (and no
BuifferedXxxputStreams) I see speeds of 11.7 MB/s reading and 11.4 writing.
Increasing the buffer size to (8*1024) bytes results in 11.7 MB/s reading and
writing. Increasing it again to (64*1024) made no significant difference. (My
MBs here are one million bytes, not 1024*1024).

The buffer size made a bigger difference to the CPU load when writing. The CPU
load was about 75% when reading using any of the three buffer sizes; but when
writing it was at about 85% for the smaller buffer, around 50% with the 8K
buffer, and about 45% with the largest buffer.

I don't know why this machine/network is getting better figures than you from a
smaller buffer. FWIW, the test machine is a 1.5GHz Celeron laptop running
WinXP Pro (sp1, not sp2). The other machine in the test is a faster Win2K
machine with a better network card and using huge buffers (so it shouldn't
affect the tests at all). The test machine was using the JDK 1.5.0_6 client
JVM. (-server gives essentially the same numbers). The LAN itself is an
unloaded 100Mbps Ethernet; there are two switched hubs between the two machines
(but I know from previous testing that they don't affect bandwidth tests
measurably).

One thing that /might/ be relevant is that neither of my test machines use
Window's built-in firewall/packet filtering, and have no other internal
fire-wall. I don't /think/ that either machines' virus checkers actively
monitor the contents of arbitrary TCP streams (I was using ports 3344 and 4433
for the tests) but it's impossible to be sure.

-- chris

Reply to this Message

JTL.zheng - 21 Dec 2006 07:48 GMT

Thank you very much for your particular explanation Chris Uppal.
As it has almost get the limit of the LAN's bandwidth and I have some
examinations to prepare, I have not modified it these weeks.
but it is used to transfer files in our LAN now, and it works well.
Thanks again for all of your help.

: )

Reply to this Message

how to improve transfering file's efficiency

Free Magazines