Hi,
Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.
Thanks
Joshua Cranmer - 06 Dec 2007 23:02 GMT
> Would anyone please suggest me a solution to send/receive a large
> byte[] (about 300.000 elements) over network? Currently, I am
> clueless.
It is hard to suggest the ideal solution when the problem is this
underspecified, but I will stick with the most generic solution:
Open up a Socket to the desired computer and port, take its inputStream,
and then use the write(byte[]) method of said stream to send the data.
If the bandwidth is insufficient, Java has built-in GZIP compression.

Signature
Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth
Daniel Pitts - 06 Dec 2007 23:06 GMT
> Hi,
>
[quoted text clipped - 3 lines]
>
> Thanks
300k isn't that large. I would suggest using an OutputStream, and
simply writing the array to that stream. If you need to break it up into
smaller chunks, OutputStream allows you to do that.

Signature
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
Lew - 07 Dec 2007 02:12 GMT
>> Hi,
>>
[quoted text clipped - 7 lines]
> simply writing the array to that stream. If you need to break it up into
> smaller chunks, OutputStream allows you to do that.
That's 300k elements - what is the size of an element?
A 1-kB element would mean a 300 MB transfer. That's an hour's transfer.
Maybe torrent?
If you aren't in a huge hurry and just want to keep your TCP/IP pipe free,
snail-mail a CD. (If it's a LAN, sneaker-net the CD.)

Signature
Lew
Arne Vajhøj - 07 Dec 2007 02:35 GMT
>>> Would anyone please suggest me a solution to send/receive a large
>>> byte[] (about 300.000 elements) over network? Currently, I am
[quoted text clipped - 5 lines]
>
> That's 300k elements - what is the size of an element?
Unless somebody tell us otherwise we are assuming that the
elements are in the array mentioned ...
Arne
Lew - 07 Dec 2007 03:00 GMT
knguyen wrote:
>>>> Would anyone please suggest me a solution to send/receive a large
>>>> byte[] (about 300.000 elements) over network? Currently, I am
Lew wrote:
>> That's 300k elements - what is the size of an element?
> Unless somebody tell us otherwise we are assuming that the
> elements are in the array mentioned ...
Oh, yeah. Duhy. Thanks. That's a millihour's transfer, ...

Signature
Lew
Daniel Pitts - 07 Dec 2007 02:36 GMT
>>> Hi,
>>>
[quoted text clipped - 16 lines]
> If you aren't in a huge hurry and just want to keep your TCP/IP pipe
> free, snail-mail a CD. (If it's a LAN, sneaker-net the CD.)
byte[] = new byte[300000]; // 300k elements.
Even at 5k/s, thats only one minute.
Even a 300MB transfer on modern connections is around 5 minutes.
In any case, opening a socket, and writing to it is probably the best
approach. If you need the ability to resume after disconnect, you might
look into using FTP or some similar protocol. Or, create your own
simple protocol.

Signature
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
Arne Vajhøj - 07 Dec 2007 00:13 GMT
> Would anyone please suggest me a solution to send/receive a large
> byte[] (about 300.000 elements) over network? Currently, I am
> clueless.
300 KB is not large.
What have you tried ?
It is relative straightforward.
Below is attached a very simple example.
Arne
=================================
import java.io.*;
import java.net.*;
public class Server {
public static void main(String[] args) throws Exception {
ServerSocket ss = new ServerSocket(9999);
Socket s = ss.accept();
InputStream is = s.getInputStream();
OutputStream os = new FileOutputStream("C:\\z2.zip");
byte[] b = new byte[10000];
int n;
while((n = is.read(b)) >= 0) {
os.write(b, 0, n);
}
os.close();
is.close();
}
}
import java.io.*;
import java.net.*;
public class Client {
public static void main(String[] args) throws Exception {
Socket s = new Socket("localhost", 9999);
OutputStream os = s.getOutputStream();
InputStream is = new FileInputStream("C:\\z1.zip");
byte[] b = new byte[10000];
int n;
while((n = is.read(b)) >= 0) {
os.write(b, 0, n);
}
os.close();
is.close();
}
}
Mark Space - 07 Dec 2007 02:12 GMT
> Hi,
>
[quoted text clipped - 3 lines]
>
> Thanks
All of the above are good. A basic socket should work fine.
Have you considered how you will test this? A basic part of software
design is proper testing. Testing works best if you design the tests
before you start.
Consider: 100T = 100 b/s = 12,500 bytes per second
300,000 / 12,500 = 24 seconds.
1,000T (giga bit) should be about 2.4 seconds.
You should expect lower numbers (as much as 3 times longer if your
network topology is lacking) but not more than about 3x. So test to
make sure your number hit this range. If they don't, try to find out why.
If you're using 10T, it might be faster to burn a CD. ;-)
Arne Vajhøj - 07 Dec 2007 02:34 GMT
> Consider: 100T = 100 b/s = 12,500 bytes per second
> 300,000 / 12,500 = 24 seconds.
[quoted text clipped - 6 lines]
>
> If you're using 10T, it might be faster to burn a CD. ;-)
????
10/100/1000 is millions bits per second not bits per second.
Arne
George Neuner - 08 Dec 2007 01:49 GMT
>Consider: 100T = 100 b/s = 12,500 bytes per second
>300,000 / 12,500 = 24 seconds.
[quoted text clipped - 6 lines]
>
>If you're using 10T, it might be faster to burn a CD. ;-)
10T = 10Mb/s = 1MB/s.
100T = 100Mb/s = 10MB/s.
1000T = 1000Mb/s = 100MB/s.
Including protocol overhead on a typical 10T Ethernet, 300KB should
take less than 1/2 second.
George
--
for email reply remove "/" from address
Mark Space - 08 Dec 2007 04:24 GMT
> 10T = 10Mb/s = 1MB/s.
> 100T = 100Mb/s = 10MB/s.
> 1000T = 1000Mb/s = 100MB/s.
Oops, yeah I think I had a brain fart there....
Roedy Green - 08 Dec 2007 06:59 GMT
On Thu, 6 Dec 2007 14:57:11 -0800 (PST), knguyen
<nguyen.h.khanh@gmail.com> wrote, quoted or indirectly quoted someone
who said :
>Would anyone please suggest me a solution to send/receive a large
>byte[] (about 300.000 elements) over network? Currently, I am
>clueless.
Use a socket and a DataOutputStream. See
http://mindprod.com/applet/fileio.html for sample code.

Signature
Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com
Arne Vajhøj - 08 Dec 2007 14:56 GMT
> On Thu, 6 Dec 2007 14:57:11 -0800 (PST), knguyen
> <nguyen.h.khanh@gmail.com> wrote, quoted or indirectly quoted someone
[quoted text clipped - 4 lines]
>
> Use a socket and a DataOutputStream.
Why ?
The service DataOutputStream provide over any OutputStream
is the ability to send items bigger than byte and handle
endianess.
Arne
Roedy Green - 08 Dec 2007 21:10 GMT
>Why ?
>
>The service DataOutputStream provide over any OutputStream
>is the ability to send items bigger than byte and handle
>endianess.
The code is compact. It does not require any parsing. It has low
overhead to prepare the stream.
I did not suggest an ObjectStream because you might overflow it with
such a long stream. It depends on the structure.
XML would be obscenely large in comparison and again might overflow
in-ram tree representations.
About your only problem with the DataOutputStream is making sure the
code on both ends is the exact same version. Unfortunately you can use
a common list of fields for both read and write the way you can in
languages such as Abundance.

Signature
Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com
Roger Lindsjö - 08 Dec 2007 22:27 GMT
>> Why ?
>>
[quoted text clipped - 15 lines]
> a common list of fields for both read and write the way you can in
> languages such as Abundance.
I read this as Arne Vajhøj meaning that the poster could just use the
regular write(byte[]) on the output stream given by the socket. Or are
you saying that the write(byte[]) has limits in the data sent (apart
from the maximum size of a byte array)?
//Roger Lindsjö
Arne Vajhøj - 09 Dec 2007 00:19 GMT
>> Why ?
>>
[quoted text clipped - 4 lines]
> The code is compact. It does not require any parsing. It has low
> overhead to prepare the stream.
It does not provide anything useful for the problem at hand
and is pure code clutter.
The OutputStream returned from the Socket has a write method that
is perfect for the task.
Arne