Any replacement for StringTokenizer/StreamTokenizer?

Thread view:

Chris Berg - 06 Nov 2005 20:47 GMT

Is there any replacement for StringTokenizer or StreamTokenizer in the
newer Java libraries? After all, they date back to Java 1.0.

I find it is difficult to fully understand and utilize
StreamTokenizer, and StringTokenizer is usually too basic to be really
competitive to parsing 'by hand' when it's a bit more complicated than
just words and whitespace.

One would expect Sun to come up with something more sophisticated and
user-friendly, as has been done in many other fields.

Chris

Reply to this Message

Thomas Hawtin - 06 Nov 2005 21:51 GMT

> Is there any replacement for StringTokenizer or StreamTokenizer in the
> newer Java libraries? After all, they date back to Java 1.0.

So does String...

java.util.regex (and Scanner) is supposed to replace them.

> I find it is difficult to fully understand and utilize
> StreamTokenizer, and StringTokenizer is usually too basic to be really
> competitive to parsing 'by hand' when it's a bit more complicated than
> just words and whitespace.

Yeah, I was looking at its use in the GTK PL&F's Metacity parser. It's
much easier to do anything non-trivial by hand.

Perhaps a full lexer should be standardised.

Tom Hawtin

Signature

Unemployed English Java programmer
http://jroller.com/page/tackline/

Reply to this Message

Roedy Green - 06 Nov 2005 22:51 GMT

On Sun, 06 Nov 2005 21:47:50 +0100, Chris Berg
<spam.spam.eggs@nd.spam> wrote, quoted or indirectly quoted someone
who said :

>One would expect Sun to come up with something more sophisticated and
>user-friendly, as has been done in many other fields.

There are two tools for more complex problems:

http://mindprod.com/jgloss/regex.html
http://mindprod.com/jgloss/parser.html

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Reply to this Message

yakovfain@gmail.com - 07 Nov 2005 00:36 GMT

String.split() is a recommended replacement for StringTokenizer. Read
the javadoc.

Yakov Fain

> Is there any replacement for StringTokenizer or StreamTokenizer in the
> newer Java libraries? After all, they date back to Java 1.0.
[quoted text clipped - 8 lines]
>
> Chris

Reply to this Message

Chris Berg - 07 Nov 2005 16:32 GMT

Hmmm... I was hoping I could somehow avoid those regular expressions.
But in vane! Regex always turns me off. I mean - whole books are
written about them (http://www.oreilly.com/catalog/regex/) - and it
was supposed to be simpler than the tokenizers ?!!? I mean, I already
have about a meter of books that I am supposed to have read.... and
then yet another one just to solve a minor parsing task?

Oh well, it was a nice try, anyway. Thanks a lot, guys... (sob) ... no
fishing trip this week-end, it seems. (now let's see... www.amazon.com
... wheres did I put that that VISA card?)

Chris

>String.split() is a recommended replacement for StringTokenizer. Read
>the javadoc.
[quoted text clipped - 13 lines]
>>
>> Chris

Reply to this Message

Roedy Green - 07 Nov 2005 16:38 GMT

On Mon, 07 Nov 2005 17:32:19 +0100, Chris Berg
<spam.spam.eggs@nd.spam> wrote, quoted or indirectly quoted someone
who said :

> I mean, I already
>have about a meter of books that I am supposed to have read.... and
>then yet another one just to solve a minor parsing task?

Just learn split. See http://mindprod.com/jgloss/regex.html

It is not hard at all.

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Reply to this Message

Any replacement for StringTokenizer/StreamTokenizer?

Free Magazines