What I’ve always feared was just proved true: Java’s StreamTokenizer does not balance quote characters. If the string is "abc'123"456"789'def", I would want the entire string to be one token, because everything is enclosed in the same pair of " quotes. What StreamTokenizer produces, however, is simply "abc'123". It ends the quoted string at the first matching quote character.
It’s also impossible to define different characters for beginning and ending quotation marks, like { for the beginning and } for the end.
Looks like I have to do quote a bit of work here myself…
