- A Concurrent Affair - https://www.concurrentaffair.org -

New Tokenizer and DrJava Properties with Attributes

As described in an earlier post [1], I am looking at all the variables an external process could need. The problem I saw was a large growth of variables that express essentially the same idea, e.g. all the open files, but that are represented differently, e.g. by using ‘:’ as separator instead of ‘ ‘.

I have now solved the problem by using attributes. Each variable can register any number of attributes and set defaults, but the user can override them. To do that, I had to write a new stream tokenizer, as the old one wasn’t easy to use when quote/brace balancing was necessary. The new one, BalancingStreamTokenizer is fairly simple; arguably simpler than the original one. There is no distinction between ordinary characters, word characters, numbers, comments, and quotes. The tokenizer always returns a string.

However, it is possible to register keywords, which are a major replacement over the single-length “ordinary character” concept of Java’s StreamTokenizer. It is furthermore possible to register multi-character quotation symbols where the beginning and the end symbol do not have to match, e.g. /* and */ or ${ and }. Inside quotes, quotation marks are balanced, so a quotation will not cut off at the first matching ending quotation mark, but rather permit correct nesting.

Here are just a few examples of what can be done now:

When the user tries to set an attribute that isn’t valid for a property, an error is emitted. The syntax highlighting works fairly well.

Now I just have to add in all the properties I mentioned earlier…

[2] [3]Share [4]