Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / December 2007

Tip: Looking for answers? Try searching our database.

send and receive a large byte[] over network?

Thread view: 
knguyen - 06 Dec 2007 22:57 GMT
Hi,

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

Thanks
Joshua Cranmer - 06 Dec 2007 23:02 GMT
> Would anyone please suggest me a solution to send/receive a large
> byte[] (about 300.000 elements) over network? Currently, I am
> clueless.

It is hard to suggest the ideal solution when the problem is this
underspecified, but I will stick with the most generic solution:

Open up a Socket to the desired computer and port, take its inputStream,
and then use the write(byte[]) method of said stream to send the data.

If the bandwidth is insufficient, Java has built-in GZIP compression.

Signature

Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth

Daniel Pitts - 06 Dec 2007 23:06 GMT
> Hi,
>
[quoted text clipped - 3 lines]
>
> Thanks

300k isn't that large.  I would suggest using an OutputStream, and
simply writing the array to that stream. If you need to break it up into
smaller chunks, OutputStream allows you to do that.
Signature

Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>

Lew - 07 Dec 2007 02:12 GMT
>> Hi,
>>
[quoted text clipped - 7 lines]
> simply writing the array to that stream. If you need to break it up into
> smaller chunks, OutputStream allows you to do that.

That's 300k elements - what is the size of an element?

A 1-kB element would mean a 300 MB transfer.  That's an hour's transfer.

Maybe torrent?

If you aren't in a huge hurry and just want to keep your TCP/IP pipe free,
snail-mail a CD.  (If it's a LAN, sneaker-net the CD.)

Signature

Lew

Arne Vajhøj - 07 Dec 2007 02:35 GMT
>>> Would anyone please suggest me a solution to send/receive a large
>>> byte[] (about 300.000 elements) over network? Currently, I am
[quoted text clipped - 5 lines]
>
> That's 300k elements - what is the size of an element?

Unless somebody tell us otherwise we are assuming that the
elements are in the array mentioned ...

Arne
Lew - 07 Dec 2007 03:00 GMT
knguyen wrote:
>>>> Would anyone please suggest me a solution to send/receive a large
>>>> byte[] (about 300.000 elements) over network? Currently, I am

Lew wrote:
>> That's 300k elements - what is the size of an element?

> Unless somebody tell us otherwise we are assuming that the
> elements are in the array mentioned ...

Oh, yeah.  Duhy.  Thanks.  That's a millihour's transfer, ...

Signature

Lew

Daniel Pitts - 07 Dec 2007 02:36 GMT
>>> Hi,
>>>
[quoted text clipped - 16 lines]
> If you aren't in a huge hurry and just want to keep your TCP/IP pipe
> free, snail-mail a CD.  (If it's a LAN, sneaker-net the CD.)

byte[] = new byte[300000]; // 300k elements.
Even at 5k/s, thats only one minute.

Even a 300MB transfer on modern connections is around 5 minutes.

In any case, opening a socket, and writing to it is probably the best
approach.  If you need the ability to resume after disconnect, you might
look into using FTP or some similar protocol.  Or, create your own
simple protocol.

Signature

Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>

Arne Vajhøj - 07 Dec 2007 00:13 GMT
> Would anyone please suggest me a solution to send/receive a large
> byte[] (about 300.000 elements) over network? Currently, I am
> clueless.

300 KB is not large.

What have you tried ?

It is relative straightforward.

Below is attached a very simple example.

Arne

=================================

import java.io.*;
import java.net.*;

public class Server {
   public static void main(String[] args) throws Exception {
      ServerSocket ss = new ServerSocket(9999);
      Socket s = ss.accept();
      InputStream is = s.getInputStream();
      OutputStream os = new FileOutputStream("C:\\z2.zip");
      byte[] b = new byte[10000];
      int n;
      while((n = is.read(b)) >= 0) {
         os.write(b, 0, n);
      }
      os.close();
      is.close();
   }
}

import java.io.*;
import java.net.*;

public class Client {
   public static void main(String[] args) throws Exception {
      Socket s = new Socket("localhost", 9999);
      OutputStream os = s.getOutputStream();
      InputStream is = new FileInputStream("C:\\z1.zip");
      byte[] b = new byte[10000];
      int n;
      while((n = is.read(b)) >= 0) {
         os.write(b, 0, n);
      }
      os.close();
      is.close();
   }
}
Mark Space - 07 Dec 2007 02:12 GMT
> Hi,
>
[quoted text clipped - 3 lines]
>
> Thanks

All of the above are good.  A basic socket should work fine.

Have you considered how you will test this?  A basic part of software
design is proper testing.  Testing works best if you design the tests
before you start.

Consider: 100T = 100 b/s = 12,500 bytes per second
300,000 / 12,500 = 24 seconds.

1,000T (giga bit) should be about 2.4 seconds.

You should expect lower numbers (as much as 3 times longer if your
network topology is lacking) but not more than about 3x.  So test to
make sure your number hit this range.  If they don't, try to find out why.

If you're using 10T, it might be faster to burn a CD. ;-)
Arne Vajhøj - 07 Dec 2007 02:34 GMT
> Consider: 100T = 100 b/s = 12,500 bytes per second
> 300,000 / 12,500 = 24 seconds.
[quoted text clipped - 6 lines]
>
> If you're using 10T, it might be faster to burn a CD. ;-)

????

10/100/1000 is millions bits per second not bits per second.

Arne
George Neuner - 08 Dec 2007 01:49 GMT
>Consider: 100T = 100 b/s = 12,500 bytes per second
>300,000 / 12,500 = 24 seconds.
[quoted text clipped - 6 lines]
>
>If you're using 10T, it might be faster to burn a CD. ;-)

10T   = 10Mb/s   = 1MB/s.
100T  = 100Mb/s  = 10MB/s.
1000T = 1000Mb/s = 100MB/s.

Including protocol overhead on a typical 10T Ethernet, 300KB should
take less than 1/2 second.

George
--
for email reply remove "/" from address
Mark Space - 08 Dec 2007 04:24 GMT
> 10T   = 10Mb/s   = 1MB/s.
> 100T  = 100Mb/s  = 10MB/s.
> 1000T = 1000Mb/s = 100MB/s.

Oops, yeah I think I had a brain fart there....
Roedy Green - 08 Dec 2007 06:59 GMT
On Thu, 6 Dec 2007 14:57:11 -0800 (PST), knguyen
<nguyen.h.khanh@gmail.com> wrote, quoted or indirectly quoted someone
who said :

>Would anyone please suggest me a solution to send/receive a large
>byte[] (about 300.000 elements) over network? Currently, I am
>clueless.

Use a socket and a DataOutputStream.  See
http://mindprod.com/applet/fileio.html for sample code.
Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Arne Vajhøj - 08 Dec 2007 14:56 GMT
> On Thu, 6 Dec 2007 14:57:11 -0800 (PST), knguyen
> <nguyen.h.khanh@gmail.com> wrote, quoted or indirectly quoted someone
[quoted text clipped - 4 lines]
>
> Use a socket and a DataOutputStream.

Why ?

The service DataOutputStream provide over any OutputStream
is the ability to send items bigger than byte and handle
endianess.

Arne
Roedy Green - 08 Dec 2007 21:10 GMT
>Why ?
>
>The service DataOutputStream provide over any OutputStream
>is the ability to send items bigger than byte and handle
>endianess.

The code is compact. It does not require any parsing. It has low
overhead to prepare the stream.

I did not suggest an ObjectStream because you might overflow it  with
such a long stream. It depends on the structure.

XML would be obscenely large in comparison and again might overflow
in-ram tree representations.

About your only problem with the DataOutputStream is making sure the
code on both ends is the exact same version. Unfortunately you can use
a common list of fields for both read and write the way you can in
languages such as Abundance.

Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Roger Lindsjö - 08 Dec 2007 22:27 GMT
>> Why ?
>>
[quoted text clipped - 15 lines]
> a common list of fields for both read and write the way you can in
> languages such as Abundance.

I read this as Arne Vajhøj meaning that the poster could just use the
regular write(byte[]) on the output stream given by the socket. Or are
you saying that the write(byte[]) has limits in the data sent (apart
from the maximum size of a byte array)?

//Roger Lindsjö
Arne Vajhøj - 09 Dec 2007 00:19 GMT
>> Why ?
>>
[quoted text clipped - 4 lines]
> The code is compact. It does not require any parsing. It has low
> overhead to prepare the stream.

It does not provide anything useful for the problem at hand
and is pure code clutter.

The OutputStream returned from the Socket has a write method that
is perfect for the task.

Arne


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.