How to identify File encoding in Java?

Thread view:

Perma - 17 Apr 2007 17:25 GMT

Hi,
I have a Java program which polls a directory for incoming files
(zipped and text).
When a new file comes, I read it and post it's outcome.

Here I have some encoding problems. The text files are usually UTF-8,
so I hard-code the encoding to UTF-8:

Code extract:
...
// trying to read the file "myFile"
FileInputStream fi = new FileInputStream(myFile);
InputStreamReader ir = new InputStreamReader(fi, "UTF8"); // hardcoded
UTF-8, how can I do this dynamically?
...

I was expecting the zipped files to be UTF-8 as well, but it turned
out not to be, so I get an:
MalformedInputException at
sun.io.ByteToCharUTF8.convert([BII[CII)I(ByteToCharUTF8

So I have to handle the two separately and it troubles my code.

I guess there's a smart way of doing this.
Hope someone can give me some hint on this! :-)

Regards, Per Magnus

Reply to this Message

Gordon Beaton - 17 Apr 2007 17:38 GMT

> I was expecting the zipped files to be UTF-8 as well, but it turned
> out not to be, so I get an:

> MalformedInputException at
> sun.io.ByteToCharUTF8.convert([BII[CII)I(ByteToCharUTF8
>
> So I have to handle the two separately and it troubles my code.

Character encoding is an attribute that describes how characters are
represented as bytes in *text* files. Classes that are "readers" read
from a stream and apply the specified encoding to obtain the text.
This operation is only meaninful when applied to text files.

Zipped files are binary, not text, so they don't contain characters
and there is no character encoding. To read a zipped file, use an
InputStream (to read raw bytes), or e.g. a ZipInputStream to get the
unzipped contents.

/gordon

--

Reply to this Message

Mike Schilling - 17 Apr 2007 20:30 GMT

> Zipped files are binary, not text, so they don't contain characters
> and there is no character encoding.

That is, zip files are binary. Zipped files (the files that were processed
to create the zip file) may be binary or text.

Reply to this Message

Kai Schwebke - 17 Apr 2007 17:40 GMT

Perma schrieb:

> I was expecting the zipped files to be UTF-8 as well, but it turned
> out not to be, so I get an:
[quoted text clipped - 5 lines]
> I guess there's a smart way of doing this.
> Hope someone can give me some hint on this! :-)

There is no 100% solution if you have to guess the encoding.
But you can exploit the fact, that input which may be interpreted
as UTF-8-encoded without error, in almost all cases is actually
UTF-8 encoded.

So just try do apply UTF-8, catch the exception or search for
the invalid marker character (defaults to \uFFFD) and apply
an alternate charset on error.

Kai

Reply to this Message

Oliver Wong - 17 Apr 2007 23:09 GMT

> Hi,
> I have a Java program which polls a directory for incoming files
[quoted text clipped - 21 lines]
> I guess there's a smart way of doing this.
> Hope someone can give me some hint on this! :-)

Maybe I'm misunderstanding something, but zip files are NOT text files
encoded via the UTF-8 encoding. In fact, they're not text files at all,
but binary files. Thus the question of "which encoding?" never has a
chance to come up at all.

- Oliver

Reply to this Message

Perma - 24 Apr 2007 15:41 GMT

I am aware of that a zip file is not a text file, but it contains a
text file.
When unzipping the text file, I need to figure out what encoding it
is.

>From the postings above, it sounds to me as if there is no method

which can tell me whether it is "UTF-8" or not, so perhaps the best
solution is to try to read the file as UTF-8, and handling non-UTF-8
files by re-reading them in the Exception handling method.
Example:

InputStreamReader ir
try {
// trying to read as utf-8
ir = new InputStreamReader(fi, "UTF8");
} catch (Exception e {
// couldn't read file as UTF-8, therefore reading it as nont-UTF-8
encoding
ir = new InputStreamReader(fi);
}

Thank you for responses!
And please update if you have some alternative solutions to this.

-Per Magnus

> > Hi,
> > I have a Java program which polls a directory for incoming files
[quoted text clipped - 28 lines]
>
> - Oliver

Reply to this Message

Martin Gregorie - 24 Apr 2007 21:32 GMT

> I am aware of that a zip file is not a text file, but it contains a
> text file. When unzipping the text file, I need to figure out what encoding it
> is.

Fire up your ZIP utility and take a good look at the file headers it
lists from the zip archive. What you see there is about all it knows
about compressed files. I don't think it knows or cares what's in the
file apart from what's implied by the filename extension.

I think you'll be better off extracting the file as a bytestream
because I'm 99% certain that's what the zip archiver thought it had
compressed.

Don't forget that ASCII text, Unicode wordprocessor files,
JPEG images and binary executables are all the same to the ZIP archiver.

Signature

martin@ | Martin Gregorie
gregorie. | Essex, UK
org |

Reply to this Message

How to identify File encoding in Java?

Free Magazines