Compressing 12 bytes?  That's a tough one.  The concern is that the
overhard added by any compression algorithm for something like the
dictionary is going to be more than 12 bytes in length.  I'd be
considering the following options:

1. Drop it.  There may be very little that you can do.

2. Run length encoding.  If you expect this be a reliable hit, then a
simple RFE scheme could be devised that only adds a byte for the
uncompressable case and helps a little with the compressable case.

3. Out-of-stream dictionary.  If you can devise a dictionary of pre-
known patterns that are likely to occur frequently in the data, then you
could pre-build that dictionary and devise a compression scheme that
asume them.

None of the three cases involve using compression libraries, though.  
You'd have to build this stuff yourself.

Consider the communication overhead of the protocol you're using.  You might
accomplish more packing multiple chunks into a message than compressing the
chunks.

On Tue, 6 Dec 2005 16:58:05 -0700, "Monique Y. Mudama"
<spam@bounceswoosh.org> wrote, quoted or indirectly quoted someone who
said :

the key to this in getting a big sample to compress and studying it
and seeing that both ends have access to the same summary info.  You
won't  do much with totally self contained messages.

What do you know about the data?  Every additional fact can be
exploited in designing custom compression.

It the ideal case, your 12 bytes are one of 100 phrases (commands) and
you can replace them with a 1 byte int indexing the message.

How much data are you actually sending at a time?  If it is only 12
bytes and you aren't sending them fast enough to get a lot of them into
a TCP packet then compression will gain you nothing.  If you are then
GZIP ought to work great.

No general-purpose compression scheme is going to be very much help for such
short strings.  They work by learning from the data, and 12 bytes doesn't give
enough input for the compressor to learn much, let alone start using what it
has learned.

So you /have/ to use a custom compression scheme.  Adding to the suggestions
that have already been made:

The gzip compression scheme (as implemented in the Java API) can be pre-trained
with representative data (which is known to both the reading and writing
applications and therefore does not need to be transmitted).  You can easily
test how well that would work for you by assembling a file consisting of (say)
8K of real, representative, data and another which is exactly the same except
that it has one more code tacked onto the end.  The difference in size between
to the two files when compressed with gzip -9 will give an indication of the
kind of compression available via this approach.  I suspect it won't work well,
but you never know and it's easy to try out.

Otherwise you will almost certainly have to look into Huffman compression or
even (if you are sufficiently desperate) Arithmetic compression (much slower
and harder to implement).

One thing to consider is whether the data has any structure.  For instance
(this is a real example from a previous job) if you desperately need to
compress URLs such as found on the web:
   http://java.sun.com/index.html
you can divide them up into separate "regions":
   http://
   java.sun
   .com/
   index
   .html
each of which has distinct frequency characteristics.  The first region, the
"scheme" can probably be represented in 1 bit in the majority of cases.  You
are looking at a population that is:
   http://                  -- the overwhelming majority
   https://                 -- quite common
   ftp://                     -- moderately rare
   <everything else>    -- rare
So you can represent http:// as one bit, probably use three bits for https://
and ftp://, and more bits than that for other cases.  Similarly the ".com"
suffix can be recognised as the most common top-level domain, and so can be
compressed to one or two bits.  The .htm and .html suffixes are common too.
You can use the Huffman algorithm to assign near optimal bit-patterns to the
various cases.  OTOH the 'java.sun' bit is going to be much more "random" so
the best you can hope for is a normal text compression scheme, such as
character-by-character Huffman compression based of the frequency of ASCII
characters in domain names (less the top-level segment).

BTW, if this data is the result of encoding some compound information, then
look at the information itself rather than the strings produced by your current
code scheme.  I.e. don't think of the task as compressing the 12-bytes of data,
but of compressing the N-bytes of information that is (currently) expressed by
the 12-bytes.

   -- chris

Hello Monique,

If your data is stream based than you can try out Huffmans Compression
algorithm. Its an algorithm that converts fixed length blocks to
variable length keys that can be used to look up the actual data. The
implementation of this algorithm may be trivial but requires some great
deal of bit shifts. You can find more info aboute Huffmans Compression
algorithm from the web. In your case is RLE fails or does not satisfies
you needs than Huffmans compression is the best bet. You can achive a
good compression ratio with it even for data blocks that are
distributed  and hence do not get compressed much in RLE.

But please bear in mind that you should avoid third party libraries as
they might add some overhead to the data being transmitted. Instead I
suggest you to approach your problem in following phase :

1. Analyze the ASCII data that is being transmitted and prepare a table
(Will be of 256 entries in case of ASCII) for each ASCII charecter and
it corrosponding frequency of occurence in your test analysis.

2. Mannually prepare the Huffman Tree from the obtained data. Hence you
will find the optimal encoding scheme for you data set. That means that
for charecter occuring most will get a shorter bit string than the
charecter that does occurs that repeatedly.

3. Implement this encoding scheme into your Tramitter / Reciever
application. You can very well implement FilterInputstream and
FilterOutputStream interfaces  into a class that will decorate your
Socket I/O stream.

I guess that above scheme will do your job as in any data some
charecters are more frequent than other.

----
VJ

Monique Y. Mudama wrote On 12/06/05 18:58,:

   Thought #1: If you're "streaming" the data, can you
treat the entire stream as the compressible block instead
of compressing each mini-block separately?

   Thought #2: Compression introduces delay, not only by
consuming CPU cycles but from the very fact that it absorbs
more bits than it emits.  E.g., if your super-efficient
compressor squeezes twelve bytes down to four bits, it
needs to swallow two twelve-byte packets before it can
transmit one compressed byte.  If you're "streaming" the
data, this might not be acceptable.

   Thought #3: If protocol overhead already amounts to
the equivalent of twelve bytes, no compressor can possibly
do better than halve your bandwidth.  If protocol overhead
is more like sixty bytes, compression cannot possibly give
you more than a one-sixth reduction.

   Thought #4: My news server carries three Usenet groups
with "compression" in their titles ...

On 2005-12-06, Monique Y. Mudama asked a question ...

Thanks all for your info and ideas.  It's now clear to me that I didn't
even know how to analyze my data, and your feedback has helped me figure
out how I will need to reframe the question.  I'm going to try to get a
better understanding of my data, rather than trying to answer questions
piecemeal and frustrating everyone.

On Tue, 6 Dec 2005 16:58:05 -0700, "Monique Y. Mudama"
<spam@bounceswoosh.org> wrote, quoted or indirectly quoted someone who
said :

Compression will buy you little if every chunk goes out in its own
packet. The packet wrapper overhead will overwhelm the payload.
See http://mindprod.com/jgloss/tcpip.html to see what I am talking
about.

However, if you your packet send delay is sufficient to say put 2
chunks most of the time in a packet you will halve the packet
overhead.

My suggestion then is to see if there is a way to tweak the send delay
to find an the optimal value for acceptable latency and fast
transmission. This will likely prove more fruitful than working on
compression.