I would rather say a class for every Token class.

I understand. On the other side it is necessary to implement somehow the
creation logic somewhere

mmm, the only solution I can think of are generics, supposing you are
using 1.5. I never used generics in Java (still using 1.4.2), so I can't
go any further, but I think that with C++ templates it should be possible.
Something like a template factory class, or a template addToken method.

Roedy Green <look-on@mindprod.com.invalid> wrote or quoted:

That really was bad, though.  At least these days machines can
read and write source code - and do some editing tasks for us.

The preprocessor was one of the things that made that *very*
challenging to do properly in C/C++.

 I could not resist that temptation.

 I have a very early version of such an experimental
 preprocessor, which I develope just for fun.

 It is written in Java and called "Metava". To simplify the
 implementation, it uses a package that can read extended
 S-expressions in a language, which I call "Unotal".

 My first goal is to have Metava to be able to pre-process
 itself to Java, so that I can write Metava in Metava.

 This is not yet finished; what works so far is that the
 following input (which is the very start of the preprocessor
 source code only) is translated to Java.

< import =
 < de.dclj.ram.notation.unotal.RoomSource
   de.dclj.ram.notation.unotal.StringValue
   de.dclj.ram.notation.unotal.SetValue
   de.dclj.ram.notation.unotal.FileRoomModule.fileRoom >

 Hello =
 < verbose         = < from=String < &ePrint it >>
   error           = < from=String < &ePrint it >>
   warning         = < from=String < &ePrint it >>
   emit            = < from=String < &write it >>
   emitLine        = < < &line > < &emitIndentation >>
   quit            = < < &verbose ["quitting"] >< &System.exit 99 >>
   indentation     = < type=int >
   indent          = << &increment indentation by=2 >>
   outdent         = << &decrement indentation by=2 >>
   emitIndentation = < < &repeat times=indentation < &write [" "] >  >>
   emitLine        = < from=int < &repeat < &line >>< &emitIndentation >>
   emitOpen        = < < &emitLine >< &emit ["{ "] >< &indent >>
   emitLeft        = < from=boolean < &emitLine >< &emit ["("] >< &if <&emit [" "]>> <&indent> >
   emitClose       = < from=boolean < &if <&emit [" "]>>< &emit ["}"]>< &outdent >>
   emitRight       = < from=boolean < &if <&emit [" "]>>< &emit [")"]>< &outdent >>
   < mainClass = < type=String value=["Hello"] >
     source = < type=RoomSource >
     < &verbose ["OK 555E6k: main"] >>>>>

 The last three lines are supposed to become the body of the
 main-method, but I see from the output, that a bug still
 prevents it to be generated from them.

 There are some fixed ad-hoc abbreviations for names used
 often, like "ePrint" for "java.lang.System.err.printLn", but I
 try to keep their number small.

 The point is that I can implement whatever notation I would
 like to have in that preprocessor. When I write a program with
 it and miss a notation, I can add it to the preprocessor and
 use it.

 To explain a procedure definition in more detail as an example:

emitLine = < from=int < &repeat < &line >>< &emitIndentation >>

 defines a procedure "emitLine". The name of the int-Parameter
 defaults to "it". The "&repeat" keyword generates a loop to
 repeat something a number of times, which - if unspecified -
 also defaults to "it". So this gives "emitLine" as it can
 be seen in the following Java-source-code that was generated
 from the Metava-Source-code above:

import de.dclj.ram.notation.unotal.RoomSource;
import de.dclj.ram.notation.unotal.StringValue;
import de.dclj.ram.notation.unotal.SetValue;
import static de.dclj.ram.notation.unotal.FileRoomModule.fileRoom;

public class
Hello
{
 public static void
 emitClose
 ( final boolean it )
 {
   if
   ( it )
   {
     emit
     ( " " );  }
   emit
   ( "}" );
   outdent
   ();  }
 
 public static void
 emitIndentation
 ()
 {
   for
   ( int i = indentation; i-- > 0; )java.lang.System.out.print
   ( " " );  }
 
 public static void
 outdent
 ()
 {
   indentation -= 2;
    }
 
 public static void
 emitLeft
 ( final boolean it )
 {
   emitLine
   ();
   emit
   ( "(" );
   if
   ( it )
   {
     emit
     ( " " );  }
   indent
   ();  }
 
 public static int indentation;
 
 public static void
 error
 ( final String it )
 {
   java.lang.System.err.println
   ( it );  }
 
 public static void
 quit
 ()
 {
   verbose
   ( "quitting" );
   System.exit
   ( 99 );  }
 
 public static void
 emit
 ( final String it )
 {
   java.lang.System.out.print
   ( it );  }
 
 public static void
 main
 ( final java.lang.String[] it )
 { }
 
 public static void
 warning
 ( final String it )
 {
   java.lang.System.err.println
   ( it );  }
 
 public static void
 emitRight
 ( final boolean it )
 {
   if
   ( it )
   {
     emit
     ( " " );  }
   emit
   ( ")" );
   outdent
   ();  }
 
 public static void
 verbose
 ( final String it )
 {
   java.lang.System.err.println
   ( it );  }
 
 
 public static void
 emitLine
 ( final int it )
 {
   for
   ( int i = it; i-- > 0; )line
   ();
   emitIndentation
   ();  }
 
 public static void
 emitLine
 ()
 {
   line
   ();
   emitIndentation
   ();  }
 
 public static void
 indent
 ()
 {
   indentation += 2;
    }
 
 public static void
 emitOpen
 ()
 {
   emitLine
   ();
   emit
   ( "{ " );
   indent
   ();  }}

Nothing's particularly stopping you from making m4 or even cpp part of
your build process.  It won't play nice with javac's automatic dependency
resolution, but that's not impossible to work around (just run m4 on the
whole source tree before doing any compilation).

Make sure you really need that before implementing such a beast.

-O

Is there a base class for the tokenizers?  That's where generic
convenience methods would go.