...

...

You were expecting 1, 2, 3, , 5 instead, right?

It seems that StringTokenizer class is working this way ...

 result, initially empty
 while ( stream is not exhausted )
 {
   append current character to result
     while ( current character is not delimiter )

   move (the pointer) to character after delimiter
     now that the current character is delimiter
 }
 result at this point has the parsed/tokenized stream

You could (easily) use LinkedList & (List)Iterator classes to build
your own version of tokenized stream w/ the desired behaviour.  The
only thing that would require more testing & planning than anything in
the project is dealing w/ the delimiter.

The change for your desired behaviour will be just before the end of
the main loop...

 result, initially empty
 while ( stream is not exhausted )
 {
   ...

   //  CHANGE
   append empty place holder to result
     if current character is again same as delimiter
 }
 result at this point has the parsed/tokenized stream

...Above presentation is quite simplistic; it will fail miserably on
escaped delimiters.  It does not handle multiple delimiters or delimiters
of length greater than 1.  (Personally, i would implement the 2d part
before i handle multiple delimiters.)

Oh, do not forget to look up StreamTokenizer too, mentioned in
StreamTokenizer's API reference (JDK 1.3).

 - parv

There is a constructor for StringTokenizer with a boolean argument
called "returnDelims". You might use that with returnDelims=true.
See the following code and output below:

 public static void main(String[] args) {
   String s = "1,2,,3,,,4";
   StringTokenizer st = new StringTokenizer(s, ",", true);

   while (st.hasMoreElements())
   {
     System.out.println("next token: " + st.nextToken());
   }

 }

next token: 1

next token: ,

next token: 2

next token: ,

next token: ,

next token: 3

next token: ,

next token: ,

next token: ,

next token: 4

Stephan
__________________________________________
Stephan Wehner
Editor, Traffic Life: Passionate Tales and Exit Strategies
An anthology about our car culture and alternatives with
short stories, poems, cartoons and lots of other art
www.trafficlife.com

There are three solutions to this problem:

1) Use a replacement StringTokenizer that allows empty tokens to be returned:
http://ostermiller.org/utils/StringTokenizer.html

2) Use the regular expression String.Split method that comes with Java 1.4 and up:
String[] tokens = "1|2|3||5".split("\\|");

3) Use a third party split library like the one here:
http://ostermiller.org/utils/StringHelper.java:
String[] tokens = StringHelper.split("1|2|3||5","|");

Stephen