1 ANTLR v3.0.1 C Runtime
\r
5 At the moment, the use of the C runtime engine for the parser is not generally
\r
6 for the inexperienced C programmer. However this is mainly because of the lack
\r
7 of documentation on use, which will be corrected shortly. The C runtime
\r
8 code itself is however well documented with doxygen style comments and a
\r
9 reasonably experienced C programmer should be able to piece it together. You
\r
10 can visit the documentation at: http://www.antlr.org/api/C/index.html
\r
12 The general make up is that everything is implemented as a pseudo class/object
\r
13 initialized with pointers to its 'member' functions and data. All objects are
\r
14 (usually) created by factories, which auto manage the memory allocation and
\r
15 release and generally make life easier. If you remember this rule, everything
\r
16 should fall in to place.
\r
18 Jim Idle - Portland Oregon, Jan 2008
\r
21 ===============================================================================
\r
23 Terence Parr, parrt at cs usfca edu
\r
24 ANTLR project lead and supreme dictator for life
\r
25 University of San Francisco
\r
29 Welcome to ANTLR v3! I've been working on this for nearly 4 years and it's
\r
30 almost ready! I plan no feature additions between this beta and first
\r
31 3.0 release. I have lots of features to add later, but this will be
\r
32 the first set. Ultimately, I need to rewrite ANTLR v3 in itself (it's
\r
33 written in 2.7.7 at the moment and also needs StringTemplate 3.0 or
\r
36 You should use v3 in conjunction with ANTLRWorks:
\r
38 http://www.antlr.org/works/index.html
\r
40 WARNING: We have bits of documentation started, but nothing super-complete
\r
41 yet. The book will be printed May 2007:
\r
43 http://www.pragmaticprogrammer.com/titles/tpantlr/index.html
\r
45 but we should have a beta PDF available on that page in Feb 2007.
\r
47 You also have the examples plus the source to guide you.
\r
49 See the new wiki FAQ:
\r
51 http://www.antlr.org/wiki/display/ANTLR3/ANTLR+v3+FAQ
\r
53 and general doc root:
\r
55 http://www.antlr.org/wiki/display/ANTLR3/ANTLR+3+Wiki+Home
\r
57 Please help add/update FAQ entries.
\r
59 I have made very little effort at this point to deal well with
\r
60 erroneous input (e.g., bad syntax might make ANTLR crash). I will clean
\r
61 this up after I've rewritten v3 in v3.
\r
63 Per the license in LICENSE.txt, this software is not guaranteed to
\r
64 work and might even destroy all life on this planet:
\r
66 THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
\r
67 IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
\r
68 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
\r
69 DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT,
\r
70 INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
\r
71 (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
\r
72 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
\r
73 HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
\r
74 STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
\r
75 IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
\r
76 POSSIBILITY OF SUCH DAMAGE.
\r
80 ANTLR v3 sample grammars:
\r
82 http://www.antlr.org/download/examples-v3.tar.gz
\r
84 contains the following examples: LL-star, cminus, dynamic-scope,
\r
85 fuzzy, hoistedPredicates, island-grammar, java, python, scopes,
\r
86 simplecTreeParser, treeparser, tweak, xmlLexer.
\r
88 Also check out Mantra Programming Language for a prototype (work in
\r
91 http://www.linguamantra.org/
\r
93 ----------------------------------------------------------------------
\r
97 ANTLR stands for (AN)other (T)ool for (L)anguage (R)ecognition and was
\r
98 originally known as PCCTS. ANTLR is a language tool that provides a
\r
99 framework for constructing recognizers, compilers, and translators
\r
100 from grammatical descriptions containing actions. Target language list:
\r
102 http://www.antlr.org/wiki/display/ANTLR3/Code+Generation+Targets
\r
104 ----------------------------------------------------------------------
\r
106 How is ANTLR v3 different than ANTLR v2?
\r
108 See migration guide:
\r
109 http://www.antlr.org/wiki/display/ANTLR3/Migrating+from+ANTLR+2+to+ANTLR+3
\r
111 ANTLR v3 has a far superior parsing algorithm called LL(*) that
\r
112 handles many more grammars than v2 does. In practice, it means you
\r
113 can throw almost any grammar at ANTLR that is non-left-recursive and
\r
114 unambiguous (same input can be matched by multiple rules); the cost is
\r
115 perhaps a tiny bit of backtracking, but with a DFA not a full parser.
\r
116 You can manually set the max lookahead k as an option for any decision
\r
117 though. The LL(*) algorithm ramps up to use more lookahead when it
\r
118 needs to and is much more efficient than normal LL backtracking. There
\r
119 is support for syntactic predicate (full LL backtracking) when LL(*)
\r
122 Lexers are much easier due to the LL(*) algorithm as well. Previously
\r
123 these two lexer rules would cause trouble because ANTLR couldn't
\r
124 distinguish between them with finite lookahead to see the decimal
\r
127 INT : ('0'..'9')+ ;
\r
128 FLOAT : INT '.' INT ;
\r
130 The syntax is almost identical for features in common, but you should
\r
131 note that labels are always '=' not ':'. So do id=ID not id:ID.
\r
133 You can do combined lexer/parser grammars again (ala PCCTS) both lexer
\r
134 and parser rules are defined in the same file. See the examples.
\r
135 Really nice. You can reference strings and characters in the grammar
\r
136 and ANTLR will generate the lexer for you.
\r
138 The attribute structure has been enhanced. Rules may have multiple
\r
139 return values, for example. Further, there are dynamically scoped
\r
140 attributes whereby a rule may define a value usable by any rule it
\r
141 invokes directly or indirectly w/o having to pass a parameter all the
\r
144 ANTLR v3 tree construction is far superior--it provides tree rewrite
\r
145 rules where the right hand side is simply the tree grammar fragment
\r
146 describing the tree you want to build:
\r
149 : typename declarator (',' typename declarator )*
\r
150 -> ^(ARG typename declarator)+
\r
153 That builds tree sequences like:
\r
155 ^(ARG int v1) ^(ARG int v2)
\r
157 ANTLR v3 also incorporates StringTemplate:
\r
159 http://www.stringtemplate.org
\r
161 just like AST support. It is useful for generating output. For
\r
162 example this rule creates a template called 'import' for each import
\r
163 definition found in the input stream:
\r
171 : 'import' identifierStar SEMI
\r
172 -> import(name={$identifierStar.st},
\r
173 begin={$identifierStar.start},
\r
174 end={$identifierStar.stop})
\r
177 The attributes are set via assignments in the argument list. The
\r
178 arguments are actions with arbitrary expressions in the target
\r
179 language. The .st label property is the result template from a rule
\r
180 reference. There is a nice shorthand in actions too:
\r
182 %foo(a={},b={},...) ctor
\r
183 %({name-expr})(a={},...) indirect template ctor reference
\r
184 %{string-expr} anonymous template from string expr
\r
185 %{expr}.y = z; template attribute y of StringTemplate-typed expr to z
\r
186 %x.y = z; set template attribute y of x (always set never get attr)
\r
187 to z [languages like python without ';' must still use the
\r
188 ';' which the code generator is free to remove during code gen]
\r
189 Same as '(x).setAttribute("y", z);'
\r
191 For ANTLR v3 I decided to make the most common tasks easy by default
\r
192 rather. This means that some of the basic objects are heavier weight
\r
193 than some speed demons would like, but they are free to pare it down
\r
194 leaving most programmers the luxury of having it "just work." For
\r
195 example, to read in some input, tweak it, and write it back out
\r
196 preserving whitespace, is easy in v3.
\r
198 The ANTLR source code is much prettier. You'll also note that the
\r
199 run-time classes are conveniently encapsulated in the
\r
200 org.antlr.runtime package.
\r
202 ----------------------------------------------------------------------
\r
204 How do I install this damn thing?
\r
206 Just untar and you'll get:
\r
208 antlr-3.0b6/README.txt (this file)
\r
209 antlr-3.0b6/LICENSE.txt
\r
210 antlr-3.0b6/src/org/antlr/...
\r
211 antlr-3.0b6/lib/stringtemplate-3.0.jar (3.0b6 needs 3.0)
\r
212 antlr-3.0b6/lib/antlr-2.7.7.jar
\r
213 antlr-3.0b6/lib/antlr-3.0b6.jar
\r
215 Then you need to add all the jars in lib to your CLASSPATH.
\r
217 ----------------------------------------------------------------------
\r
219 How do I use ANTLR v3?
\r
221 [I am assuming you are only using the command-line (and not the
\r
224 Running ANTLR with no parameters shows you:
\r
226 ANTLR Parser Generator Early Access Version 3.0b6 (Jan 31, 2007) 1989-2007
\r
227 usage: java org.antlr.Tool [args] file.g [file2.g file3.g ...]
\r
228 -o outputDir specify output directory where all output is generated
\r
229 -lib dir specify location of token files
\r
230 -report print out a report about the grammar(s) processed
\r
231 -print print out the grammar without actions
\r
232 -debug generate a parser that emits debugging events
\r
233 -profile generate a parser that computes profiling information
\r
234 -nfa generate an NFA for each rule
\r
235 -dfa generate a DFA for each decision point
\r
236 -message-format name specify output style for messages
\r
237 -X display extended argument list
\r
239 For example, consider how to make the LL-star example from the examples
\r
240 tarball you can get at http://www.antlr.org/download/examples-v3.tar.gz
\r
242 $ cd examples/java/LL-star
\r
243 $ java org.antlr.Tool simplec.g
\r
251 int foo(int y, char d) {
\r
253 for (i=0; i<3; i=i+1) {
\r
259 you will see output as follows:
\r
262 bar is a declaration
\r
263 foo is a definition
\r
265 What if I want to test my parser without generating code? Easy. Just
\r
266 run ANTLR in interpreter mode. It can't execute your actions, but it
\r
267 can create a parse tree from your input to show you how it would be
\r
268 matched. Use the org.antlr.tool.Interp main class. In the following,
\r
269 I interpret simplec.g on t.c, which contains "int x;"
\r
271 $ java org.antlr.tool.Interp simplec.g WS program t.c
\r
272 ( <grammar SimpleC>
\r
276 ( type [@0,0:2='int',<14>,1:0] )
\r
277 ( declarator [@2,4:4='x',<2>,1:4] )
\r
278 [@3,5:5=';',<5>,1:5]
\r
284 where I have formatted the output to make it more readable. I have
\r
285 told it to ignore all WS tokens.
\r
287 ----------------------------------------------------------------------
\r
289 How do I rebuild ANTLR v3?
\r
291 Make sure the following two jars are in your CLASSPATH
\r
293 antlr-3.0b6/lib/stringtemplate-3.0.jar
\r
294 antlr-3.0b6/lib/antlr-2.7.7.jar
\r
295 junit.jar [if you want to build the test directories]
\r
297 then jump into antlr-3.0b6/src directory and then type:
\r
299 $ javac -d . org/antlr/Tool.java org/antlr/*/*.java org/antlr/*/*/*.java
\r
301 Takes 9 seconds on my 1Ghz laptop or 4 seconds with jikes. Later I'll
\r
302 have a real build mechanism, though I must admit the one-liner appeals
\r
303 to me. I use Intellij so I never type anything actually to build.
\r
305 There is also an ANT build.xml file, but I know nothing of ANT; contributed
\r
306 by others (I'm opposed to any tool with an XML interface for Humans).
\r
308 -----------------------------------------------------------------------
\r
311 1. Auto-generated lexers do not inherit parent parser's @namespace
\r
312 {...} value. Use @lexer::namespace{...}.
\r
314 -----------------------------------------------------------------------
\r
320 * Jonathan DeKlotz updated C# templates to be 3.0b6 current
\r
324 * Manually-specified (...)=> force backtracking eval of that predicate.
\r
325 backtracking=true mode does not however. Added unit test.
\r
329 * Fixed bug in lexer where ~T didn't compute the set from rule T.
\r
331 * Added -Xnoinlinedfa make all DFA with tables; no inline prediction with IFs
\r
333 * Fixed http://www.antlr.org:8888/browse/ANTLR-80.
\r
334 Sem pred states didn't define lookahead vars.
\r
336 * Fixed http://www.antlr.org:8888/browse/ANTLR-91.
\r
337 When forcing some acyclic DFA to be state tables, they broke.
\r
338 Forcing all DFA to be state tables should give same results.
\r
342 * setTokenSource in CommonTokenStream didn't clear tokens list.
\r
343 setCharStream calls reset in Lexer.
\r
345 * Altered -depend. No longer printing grammar files for multiple input
\r
346 files with -depend. Doesn't show T__.g temp file anymore. Added
\r
347 TLexer.tokens. Added .h files if defined.
\r
351 * Added -depend command-line option that, instead of processing files,
\r
352 it shows you what files the input grammar(s) depend on and what files
\r
353 they generate. For combined grammar T.g:
\r
355 $ java org.antlr.Tool -depend T.g
\r
363 Now, assuming U.g is a tree grammar ref'd T's tokens:
\r
365 $ java org.antlr.Tool -depend T.g U.g
\r
374 Handles spaces by escaping them. Pays attention to -o, -fo and -lib.
\r
375 Dir 'x y' is a valid dir in current dir.
\r
377 $ java org.antlr.Tool -depend -lib /usr/local/lib -o 'x y' T.g U.g
\r
378 x\ y/TParser.java : T.g
\r
379 x\ y/T.tokens : T.g
\r
381 U.g: /usr/local/lib/T.tokens
\r
383 x\ y/U.tokens : U.g
\r
385 You have API access via org.antlr.tool.BuildDependencyGenerator class:
\r
386 getGeneratedFileList(), getDependenciesFileList(). You can also access
\r
387 the output template: getDependencies(). The file
\r
388 org/antlr/tool/templates/depend.stg contains the template. You can
\r
389 modify as you want. File objects go in so you can play with path etc...
\r
393 * no more .gl files generated. All .g all the time.
\r
395 * changed @finally to be @after and added a finally clause to the
\r
396 exception stuff. I also removed the superfluous "exception"
\r
397 keyword. Here's what the new syntax looks like:
\r
400 @after { System.out.println("ick"); }
\r
403 catch[RecognitionException e] { System.out.println("foo"); }
\r
404 catch[IOException e] { System.out.println("io"); }
\r
405 finally { System.out.println("foobar"); }
\r
407 @after executes after bookkeeping to set $rule.stop, $rule.tree but
\r
408 before scopes pop and any memoization happens. Dynamic scopes and
\r
409 memoization are still in generated finally block because they must
\r
410 exec even if error in rule. The @after action and tree setting
\r
411 stuff can technically be skipped upon syntax error in rule. [Later
\r
412 we might add something to finally to stick an ERROR token in the
\r
413 tree and set the return value.] Sequence goes: set $stop, $tree (if
\r
414 any), @after (if any), pop scopes (if any), memoize (if needed),
\r
415 grammar finally clause. Last 3 are in generated code's finally
\r
418 3.0b6 - January 31, 2007
\r
422 * Fixed bug in IntervalSet.and: it returned the same empty set all the time
\r
423 rather than new empty set. Code altered the same empty set.
\r
425 * Made analysis terminate faster upon a decision that takes too long;
\r
426 it seemed to keep doing work for a while. Refactored some names
\r
427 and updated comments. Also made it terminate when it realizes it's
\r
428 non-LL(*) due to recursion. just added terminate conditions to loop
\r
431 * Sometimes fatal non-LL(*) messages didn't appear; instead you got
\r
432 "antlr couldn't analyze", which is actually untrue. I had the
\r
433 order of some prints wrong in the DecisionProbe.
\r
435 * The code generator incorrectly detected when it could use a fixed,
\r
436 acyclic inline DFA (i.e., using an IF). Upon non-LL(*) decisions
\r
437 with predicates, analysis made cyclic DFA. But this stops
\r
438 the computation detecting whether they are cyclic. I just added
\r
439 a protection in front of the acyclic DFA generator to avoid if
\r
440 non-LL(*). Updated comments.
\r
444 * Made tree node streams use adaptor to create navigation nodes.
\r
445 Thanks to Emond Papegaaij.
\r
449 * Added lexer rule properties: start, stop
\r
453 * analysis failsafe is back on; if a decision takes too long, it bails out
\r
458 * += labels for rules only work for output option; previously elements
\r
459 of list were the return value structs, but are now either the tree or
\r
460 StringTemplate return value. You can label different rules now
\r
465 * Allow \" to work correctly in "..." template.
\r
469 * errors that are now warnings: missing AST label type in trees.
\r
470 Also "no start rule detected" is warning.
\r
472 * tree grammars also can do rewrite=true for output=template.
\r
473 Only works for alts with single node or tree as alt elements.
\r
474 If you are going to use $text in a tree grammar or do rewrite=true
\r
475 for templates, you must use in your main:
\r
477 nodes.setTokenStream(tokens);
\r
479 * You get a warning for tree grammars that do rewrite=true and
\r
480 output=template and have -> for alts that are not simple nodes
\r
481 or simple trees. new unit tests in TestRewriteTemplates at end.
\r
485 * Error message appears when you use -> in tree grammar with
\r
486 output=template and rewrite=true for alt that is not simple
\r
489 * no more $stop attribute for tree parsers; meaningless/useless.
\r
490 Removed from TreeRuleReturnScope also.
\r
492 * rule text attribute in tree parser must pull from token buffer.
\r
493 Makes no sense otherwise. added getTokenStream to TreeNodeStream
\r
494 so rule $text attr works. CommonTreeNodeStream etc... now let
\r
495 you set the token stream so you can access later from tree parser.
\r
496 $text is not well-defined for rules like
\r
500 because stat is not a single node nor rooted with a single node.
\r
501 $slist.text will get only first stat. I need to add a warning about
\r
504 * Fixed http://www.antlr.org:8888/browse/ANTLR-76 for Java.
\r
505 Enhanced TokenRewriteStream so it accepts any object; converts
\r
506 to string at last second. Allows you to rewrite with StringTemplate
\r
509 * added rewrite option that makes -> template rewrites do replace ops for
\r
510 TokenRewriteStream input stream. In output=template and rewrite=true mode
\r
511 same as before 'cept that the parser does
\r
513 ((TokenRewriteStream)input).replace(
\r
514 ((Token)retval.start).getTokenIndex(),
\r
515 input.LT(-1).getTokenIndex(),
\r
518 after each rewrite so that the input stream is altered. Later refs to
\r
519 $text will have rewrites. Here's a sample test program for grammar Rew.
\r
521 FileReader groupFileR = new FileReader("Rew.stg");
\r
522 StringTemplateGroup templates = new StringTemplateGroup(groupFileR);
\r
523 ANTLRInputStream input = new ANTLRInputStream(System.in);
\r
524 RewLexer lexer = new RewLexer(input);
\r
525 TokenRewriteStream tokens = new TokenRewriteStream(lexer);
\r
526 RewParser parser = new RewParser(tokens);
\r
527 parser.setTemplateLib(templates);
\r
529 System.out.println(tokens.toString());
\r
530 groupFileR.close();
\r
534 * BaseTree.dupTree didn't dup recursively.
\r
538 * Cleaned up some comments and removed field treeNode
\r
539 from MismatchedTreeNodeException class. It is "node" in
\r
540 RecognitionException.
\r
542 * Changed type from Object to BitSet for expecting fields in
\r
543 MismatchedSetException and MismatchedNotSetException
\r
545 * Cleaned up error printing in lexers and the messages that it creates.
\r
547 * Added this to TreeAdaptor:
\r
548 /** Return the token object from which this node was created.
\r
549 * Currently used only for printing an error message.
\r
550 * The error display routine in BaseRecognizer needs to
\r
551 * display where the input the error occurred. If your
\r
552 * tree of limitation does not store information that can
\r
553 * lead you to the token, you can create a token filled with
\r
554 * the appropriate information and pass that back. See
\r
555 * BaseRecognizer.getErrorMessage().
\r
557 public Token getToken(Object t);
\r
561 * made BaseRecognizer.displayRecognitionError nonstatic so people can
\r
562 override it. Not sure why it was static before.
\r
564 * Removed state/decision message that comes out of no
\r
565 viable alternative exceptions, as that was too much.
\r
566 removed the decision number from the early exit exception
\r
567 also. During development, you can simply override
\r
568 displayRecognitionError from BaseRecognizer to add the stuff
\r
569 back in if you want.
\r
571 * made output go to an output method you can override: emitErrorMessage()
\r
573 * general cleanup of the error emitting code in BaseRecognizer. Lots
\r
574 more stuff you can override: getErrorHeader, getTokenErrorDisplay,
\r
575 emitErrorMessage, getErrorMessage.
\r
579 * Altered Tree.Parser.matchAny() so that it skips entire trees if
\r
580 node has children otherwise skips one node. Now this works to
\r
581 skip entire body of function if single-rooted subtree:
\r
582 ^(FUNC name=ID arg=ID .)
\r
584 * Added "reverse index" from node to stream index. Override
\r
585 fillReverseIndex() in CommonTreeNodeStream if you want to change.
\r
586 Use getNodeIndex(node) to find stream index for a specific tree node.
\r
587 See getNodeIndex(), reverseIndex(Set tokenTypes),
\r
588 reverseIndex(int tokenType), fillReverseIndex(). The indexing
\r
589 costs time and memory to fill, but pulling stuff out will be lots
\r
590 faster as it can jump from a node ptr straight to a stream index.
\r
592 * Added TreeNodeStream.get(index) to make it easier for interpreters to
\r
593 jump around in tree node stream.
\r
595 * New CommonTreeNodeStream buffers all nodes in stream for fast jumping
\r
596 around. It now has push/pop methods to invoke other locations in
\r
597 the stream for building interpreters.
\r
599 * Moved CommonTreeNodeStream to UnBufferedTreeNodeStream and removed
\r
600 Iterator implementation. moved toNodesOnlyString() to TestTreeNodeStream
\r
602 * [BREAKS ANY TREE IMPLEMENTATION]
\r
603 made CommonTreeNodeStream work with any tree node type. TreeAdaptor
\r
604 now implements isNil so must add; trivial, but does break back
\r
609 * Added traceIn/Out methods to recognizers so that you can override them;
\r
610 previously they were in-line print statements. The message has also
\r
611 been slightly improved.
\r
613 * Factored BuildParseTree into debug package; cleaned stuff up. Fixed
\r
618 * [BREAKS ANY TREE IMPLEMENTATION]
\r
619 org.antlr.runtime.tree.Tree; needed to add get/set for token start/stop
\r
620 index so CommonTreeAdaptor can assume Tree interface not CommonTree
\r
621 implementation. Otherwise, no way to create your own nodes that satisfy
\r
622 Tree because CommonTreeAdaptor was doing
\r
624 public int getTokenStartIndex(Object t) {
\r
625 return ((CommonTree)t).startIndex;
\r
630 /** What is the smallest token index (indexing from 0) for this node
\r
631 * and its children?
\r
633 int getTokenStartIndex();
\r
635 void setTokenStartIndex(int index);
\r
637 /** What is the largest token index (indexing from 0) for this node
\r
638 * and its children?
\r
640 int getTokenStopIndex();
\r
642 void setTokenStopIndex(int index);
\r
646 * Added org.antlr.runtime.tree.DOTTreeGenerator so you can generate DOT
\r
647 diagrams easily from trees.
\r
649 CharStream input = new ANTLRInputStream(System.in);
\r
650 TLexer lex = new TLexer(input);
\r
651 CommonTokenStream tokens = new CommonTokenStream(lex);
\r
652 TParser parser = new TParser(tokens);
\r
653 TParser.e_return r = parser.e();
\r
654 Tree t = (Tree)r.tree;
\r
655 System.out.println(t.toStringTree());
\r
656 DOTTreeGenerator gen = new DOTTreeGenerator();
\r
657 StringTemplate st = gen.toDOT(t);
\r
658 System.out.println(st);
\r
660 * Changed the way mark()/rewind() work in CommonTreeNode stream to mirror
\r
661 more flexible solution in ANTLRStringStream. Forgot to set lastMarker
\r
662 anyway. Now you can rewind to non-most-recent marker.
\r
666 * Temp lexer now end in .gl (T__.gl, for example)
\r
668 * TreeParser suffix no longer generated for tree grammars
\r
670 * Defined reset for lexer, parser, tree parser; rewinds the input stream also
\r
674 * Made Grammar.abortNFAToDFAConversion() abort in middle of a DFA.
\r
678 * fixed bug in OrderedHashSet.add(). It didn't track elements correctly.
\r
682 * updated build.xml for future Ant compatibility, thanks to Matt Benson.
\r
684 * various tests in TestRewriteTemplate and TestSyntacticPredicateEvaluation
\r
685 were using the old 'channel' vs. new '$channel' notation.
\r
686 TestInterpretedParsing didn't pick up an earlier change to CommonToken.
\r
687 Reported by Matt Benson.
\r
689 * fixed platform dependent test failures in TestTemplates, supplied by Matt
\r
694 * optimized semantic predicate evaluation so that p||!p yields true.
\r
698 * fixed bug that prevented var = $rule.some_retval from working in anything
\r
699 but the first alternative of a rule or subrule.
\r
701 * attribute names containing digits were not allowed, this is now fixed,
\r
702 allowing attributes like 'name1' but not '1name1'.
\r
706 * Removed LeftRecursionMessage and apparatus because it seems that I check
\r
707 for left recursion upfront before analysis and everything gets specified as
\r
708 recursion cycles at this point.
\r
712 * TokenRewriteStream.replace was not passing programName to next method.
\r
716 * updated DOT files for DFA generation to make smaller circles.
\r
718 * made epsilon edges italics in the NFA diagrams.
\r
720 3.0b5 - November 15, 2006
\r
722 The biggest thing is that your grammar file names must match the grammar name
\r
723 inside (your generated class names will also be different) and we use
\r
724 $channel=HIDDEN now instead of channel=99 inside lexer actions.
\r
725 Should be compatible other than that. Please look at complete list of
\r
730 * Force token index to be -1 for CommonIndex in case not set.
\r
734 * getUniqueID for TreeAdaptor now uses identityHashCode instead of hashCode.
\r
738 * No grammar nondeterminism warning now when wildcard '.' is final alt.
\r
748 : '//' (options {greedy=false;} : .)* '\r'? '\n'
\r
752 : '//' (options {greedy=false;} : 'x'|.)* '\r'? '\n'
\r
758 * Syntactic predicates did not get hoisting properly upon non-LL(*) decision. Other hoisting issues fixed. Cleaned up code.
\r
760 * Removed failsafe that check to see if I'm spending too much time on a single DFA; I don't think we need it anymore.
\r
764 * $text, $line, etc... were not working in assignments. Fixed and added
\r
767 * $label.text translated to label.getText in lexer even if label was on a char
\r
771 * Added error if you don't specify what the AST type is; actions in tree
\r
772 grammar won't work without it.
\r
776 a : ID {String s = $ID.text;} ;
\r
778 ANTLR Parser Generator Early Access Version 3.0b5 (??, 2006) 1989-2006
\r
779 error: x.g:0:0: (152) tree grammar x has no ASTLabelType option
\r
783 * $text, $line, etc... were not working properly within lexer rule.
\r
787 * Finally actions now execute before dynamic scopes are popped it in the
\r
788 rule. Previously was not possible to access the rules scoped variables
\r
789 in a finally action.
\r
793 * Altered ActionTranslator to emit errors on setting read-only attributes
\r
794 such as $start, $stop, $text in a rule. Also forbid setting any attributes
\r
795 in rules/tokens referenced by a label or name.
\r
796 Setting dynamic scopes's attributes and your own parameter attributes
\r
801 * Altered how ANTLR figures out what decision is associated with which
\r
802 block of grammar. Makes ANTLRWorks correctly find DFA for a block.
\r
806 * Fixed bug where EOT transitions led to no NFA configs in a DFA state,
\r
807 yielding an error in DFA table generation.
\r
809 * renamed action.g to ActionTranslator.g
\r
810 the ActionTranslator class is now called ActionTranslatorLexer, as ANTLR
\r
811 generates this classname now. Fixed rest of codebase accordingly.
\r
813 * added rules recognizing setting of scopes' attributes to ActionTranslator.g
\r
814 the Objective C target needed access to the right-hand side of the assignment
\r
815 in order to generate correct code
\r
817 * changed ANTLRCore.sti to reflect the new mandatory templates to support the above
\r
818 namely: scopeSetAttributeRef, returnSetAttributeRef and the ruleSetPropertyRef_*
\r
819 templates, with the exception of ruleSetPropertyRef_text. we cannot set this attribute
\r
823 * Fixed 2 bugs in DFA conversion that caused exceptions.
\r
824 altered functionality of getMinElement so it ignores elements<0.
\r
828 * moved resetStateNumbersToBeContiguous() to after issuing of warnings;
\r
829 an internal error in that routine should make more sense as issues
\r
830 with decision will appear first.
\r
832 * fixed cut/paste bug I introduced when fixed EOF in min/max
\r
833 bug. Prevented C grammar from working briefly.
\r
837 * Removed a failsafe that seems to be unnecessary that ensure DFA didn't
\r
838 get too big. It was resulting in some failures in code generation that
\r
839 led me on quite a strange debugging trip.
\r
843 * Use channel=HIDDEN not channel=99 to put tokens on hidden channel.
\r
847 * ANTLR now has a customizable message format for errors and warnings,
\r
848 to make it easier to fulfill requirements by IDEs and such.
\r
849 The format to be used can be specified via the '-message-format name'
\r
850 command line switch. The default for name is 'antlr', also available
\r
851 at the moment is 'gnu'. This is done via StringTemplate, for details
\r
852 on the requirements look in org/antlr/tool/templates/messages/formats/
\r
854 * line numbers for lexers in combined grammars are now reported correctly.
\r
858 * ANTLRReaderStream improperly checked for end of input.
\r
862 * For ANTLRStringStream, LA(-1) was off by one...gave you LA(-2).
\r
864 3.0b4 - August 24, 2006
\r
866 * error when no rules in grammar. doesn't crash now.
\r
868 * Token is now an interface.
\r
870 * remove dependence on non runtime classes in runtime package.
\r
872 * filename and grammar name must be same Foo in Foo.g. Generates FooParser,
\r
873 FooLexer, ... Combined grammar Foo generates Foo$Lexer.g which generates
\r
874 FooLexer.java. tree grammars generate FooTreeParser.java
\r
878 * added C# target to lib, codegen, templates
\r
882 * added tree arg to navigation methods in treeadaptor
\r
886 * fixed bug related to (a|)+ on end of lexer rules. crashed instead
\r
889 * added warning that interpreter doesn't do synpreds yet
\r
891 * allow different source of classloader:
\r
892 ClassLoader cl = Thread.currentThread().getContextClassLoader();
\r
894 cl = this.getClass().getClassLoader();
\r
900 * compressed DFA edge tables significantly. All edge tables are
\r
901 unique. The transition table can reuse arrays. Look like this now:
\r
903 public static readonly DFA30_transition0 =
\r
904 new short[] { 46, 46, -1, 46, 46, -1, -1, -1, -1, -1, -1, -1,...};
\r
905 public static readonly DFA30_transition1 =
\r
906 new short[] { 21 };
\r
907 public static readonly short[][] DFA30_transition = {
\r
914 * If you defined both a label like EQ and '=', sometimes the '=' was
\r
915 used instead of the EQ label.
\r
917 * made headerFile template have same arg list as outputFile for consistency
\r
919 * outputFile, lexer, genericParser, parser, treeParser templates
\r
920 reference cyclicDFAs attribute which was no longer used after I
\r
921 started the new table-based DFA. I made cyclicDFADescriptors
\r
922 argument to outputFile and headerFile (only). I think this is
\r
923 correct as only OO languages will want the DFA in the recognizer.
\r
924 At the top level, C and friends can use it. Changed name to use
\r
925 cyclicDFAs again as it's a better name probably. Removed parameter
\r
926 from the lexer, ... For example, my parser template says this now:
\r
928 <cyclicDFAs:cyclicDFA()> <! dump tables for all DFA !>
\r
930 * made all token ref token types go thru code gen's
\r
931 getTokenTypeAsTargetLabel()
\r
933 * no more computing DFA transition tables for acyclic DFA.
\r
937 * fixed a place where I was adding syn predicates into rewrite stuff.
\r
939 * turned off invalid token index warning in AW support; had a problem.
\r
941 * bad location event generated with -debug for synpreds in autobacktrack mode.
\r
945 * changed runtime.DFA so that it treats all chars and token types as
\r
946 char (unsigned 16 bit int). -1 becomes '\uFFFF' then or 65535.
\r
948 * changed MAX_STATE_TRANSITIONS_FOR_TABLE to be 65534 by default
\r
949 now. This means that all states can use a table to do transitions.
\r
951 * was not making synpreds on (C)* type loops with backtrack=true
\r
953 * was copying tree stuff and actions into synpreds with backtrack=true
\r
955 * was making synpreds on even single alt rules / blocks with backtrack=true
\r
957 3.0b3 - July 21, 2006
\r
959 * ANTLR fails to analyze complex decisions much less frequently. It
\r
960 turns out that the set of decisions for which ANTLR fails (times
\r
961 out) is the same set (so far) of non-LL(*) decisions. Morever, I'm
\r
962 able to detect this situation quickly and report rather than timing
\r
963 out. Errors look like:
\r
965 java.g:468:23: [fatal] rule concreteDimensions has non-LL(*)
\r
966 decision due to recursive rule invocations in alts 1,2. Resolve
\r
967 by left-factoring or using syntactic predicates with fixed k
\r
968 lookahead or use backtrack=true option.
\r
970 This message only appears when k=*.
\r
972 * Shortened no viable alt messages to not include decision
\r
975 [compilationUnit, declaration]: line 8:8 decision=<<67:1: declaration
\r
976 : ( ( fieldDeclaration )=> fieldDeclaration | ( methodDeclaration )=>
\r
977 methodDeclaration | ( constructorDeclaration )=>
\r
978 constructorDeclaration | ( classDeclaration )=> classDeclaration | (
\r
979 interfaceDeclaration )=> interfaceDeclaration | ( blockDeclaration )=>
\r
980 blockDeclaration | emptyDeclaration );>> state 3 (decision=14) no
\r
981 viable alt; token=[@1,184:187='java',<122>,8:8]
\r
983 too long and hard to read.
\r
987 * Code gen bug: states with no emanating edges were ignored by ST.
\r
988 Now an empty list is used.
\r
990 * Added grammar parameter to recognizer templates so they can access
\r
991 properties like getName(), ...
\r
995 * Fixed the gated pred merged state bug. Added unit test.
\r
997 * added new method to Target: getTokenTypeAsTargetLabel()
\r
1001 * I was doing an AND instead of OR in the gated predicate stuff.
\r
1002 Thanks to Stephen Kou!
\r
1004 * Reduce op for combining predicates was insanely slow sometimes and
\r
1005 didn't actually work well. Now it's fast and works.
\r
1007 * There is a bug in merging of DFA stop states related to gated
\r
1008 preds...turned it off for now.
\r
1010 3.0b2 - July 5, 2006
\r
1014 * token emission not properly protected in lexer filter mode.
\r
1016 * EOT, EOT DFA state transition tables should be init'd to -1 (only
\r
1017 was doing this for compressed tables). Fixed.
\r
1019 * in trace mode, exit method not shown for memoized rules
\r
1021 * added -Xmaxdfaedges to allow you to increase number of edges allowed
\r
1022 for a single DFA state before it becomes "special" and can't fit in
\r
1025 * Bug in tables. Short are signed so min/max tables for DFA are now
\r
1030 * Added a method to reset the tool error state for current thread.
\r
1031 See ErrorManager.java
\r
1033 * [Got this working properly today] backtrack mode that let's you type
\r
1034 in any old crap and ANTLR will backtrack if it can't figure out what
\r
1035 you meant. No errors are reported by antlr during analysis. It
\r
1036 implicitly adds a syn pred in front of every production, using them
\r
1037 only if static grammar LL(*) analysis fails. Syn pred code is not
\r
1038 generated if the pred is not used in a decision.
\r
1040 This is essentially a rapid prototyping mode.
\r
1042 * Added backtracking report to the -report option
\r
1044 * Added NFA->DFA conversion early termination report to the -report option
\r
1046 * Added grammar level k and backtrack options to -report
\r
1048 * Added a dozen unit tests to test autobacktrack NFA construction.
\r
1050 * If you are using filter mode, you must manually use option
\r
1055 * Added k=* option so you can set k=2, for example, on whole grammar,
\r
1056 but an individual decision can be LL(*).
\r
1058 * memoize option for grammars, rules, blocks. Remove -nomemo cmd-line option
\r
1060 * but in DOT generator for DFA; fixed.
\r
1062 * runtime.DFA reported errors even when backtracking
\r
1066 * Added -X option list to help
\r
1068 * Syn preds were being hoisted into other rules, causing lots of extra
\r
1073 * unnecessary files removed during build.
\r
1075 * Matt Benson updated build.xml
\r
1077 * Detecting use of synpreds in analysis now instead of codegen. In
\r
1078 this way, I can avoid analyzing decisions in synpreds for synpreds
\r
1079 not used in a DFA for a real rule. This is used to optimize things
\r
1080 for backtrack option.
\r
1082 * Code gen must add _fragment or whatever to end of pred name in
\r
1083 template synpredRule to avoid having ANTLR know anything about
\r
1086 * Added -IdbgST option to emit ST delimiters at start/stop of all
\r
1087 templates spit out.
\r
1091 * Tweaked message when ANTLR cannot handle analysis.
\r
1093 3.0b1 - June 27, 2006
\r
1097 * syn preds no longer generate little static classes; they also don't
\r
1098 generate a whole bunch of extra crap in the rules built to test syn
\r
1099 preds. Removed GrammarFragmentPointer class from runtime.
\r
1103 * added output option to -report output.
\r
1105 * added profiling info:
\r
1106 Number of rule invocations in "guessing" mode
\r
1107 number of rule memoization cache hits
\r
1108 number of rule memoization cache misses
\r
1110 * made DFA DOT diagrams go left to right not top to bottom
\r
1112 * I try to recursive overflow states now by resolving these states
\r
1113 with semantic/syntactic predicates if they exist. The DFA is then
\r
1114 deterministic rather than simply resolving by choosing first
\r
1115 nondeterministic alt. I used to generated errors:
\r
1117 ~/tmp $ java org.antlr.Tool -dfa t.g
\r
1118 ANTLR Parser Generator Early Access Version 3.0b2 (July 5, 2006) 1989-2006
\r
1119 t.g:2:5: Alternative 1: after matching input such as A A A A A decision cannot predict what comes next due to recursion overflow to b from b
\r
1120 t.g:2:5: Alternative 2: after matching input such as A A A A A decision cannot predict what comes next due to recursion overflow to b from b
\r
1122 Now, I uses predicates if available and emits no warnings.
\r
1124 * made sem preds share accept states. Previously, multiple preds in a
\r
1125 decision forked new accepts each time for each nondet state.
\r
1129 * Need parens around the prediction expressions in templates.
\r
1131 * Referencing $ID.text in an action forced bad code gen in lexer rule ID.
\r
1133 * Fixed a bug in how predicates are collected. The definition of
\r
1134 "last predicated alternative" was incorrect in the analysis. Further,
\r
1135 gated predicates incorrectly missed a case where an edge should become
\r
1136 true (a tautology).
\r
1138 * Removed an unnecessary input.consume() reference in the runtime/DFA class.
\r
1142 * -> ($rulelabel)? didn't generate proper code for ASTs.
\r
1144 * bug in code gen (did not compile)
\r
1148 Problem is repeated ref to ID from left side. Juergen pointed this out.
\r
1150 * use of tokenVocab with missing file yielded exception
\r
1152 * (A|B)=> foo yielded an exception as (A|B) is a set not a block. Fixed.
\r
1154 * Didn't set ID1= and INT1= for this alt:
\r
1155 | ^(ID INT+ {System.out.print(\"^(\"+$ID+\" \"+$INT+\")\");})
\r
1157 * Fixed so repeated dangling state errors only occur once like:
\r
1158 t.g:4:17: the decision cannot distinguish between alternative(s) 2,1 for at least one input sequence
\r
1160 * tracking of rule elements was on (making list defs at start of
\r
1161 method) with templates instead of just with ASTs. Turned off.
\r
1163 * Doesn't crash when you give it a missing file now.
\r
1165 * -report: add output info: how many LL(1) decisions.
\r
1169 * ^(ROOT ID?) Didn't work; nor did any other nullable child list such as
\r
1170 ^(ROOT ID* INT?). Now, I check to see if child list is nullable using
\r
1171 Grammar.LOOK() and, if so, I generate an "IF lookahead is DOWN" gate
\r
1172 around the child list so the whole thing is optional.
\r
1174 * Fixed a bug in LOOK that made it not look through nullable rules.
\r
1176 * Using AST suffixes or -> rewrite syntax now gives an error w/o a grammar
\r
1177 output option. Used to crash ;)
\r
1179 * References to EOF ended up with improper -1 refs instead of EOF in output.
\r
1181 * didn't warn of ambig ref to $expr in rewrite; fixed.
\r
1183 : '[' expr 'for' type ID 'in' expr ']'
\r
1184 -> comprehension(expr={$expr.st},type={},list={},i={})
\r
1189 * EOF works in the parser as a token name.
\r
1191 * Rule b:(A B?)*; didn't display properly in AW due to the way ANTLR
\r
1194 * "scope x;" in a rule for unknown x gives no error. Fixed. Added unit test.
\r
1196 * Label type for refs to start/stop in tree parser and other parsers were
\r
1197 not used. Lots of casting. Ick. Fixed.
\r
1199 * couldn't refer to $tokenlabel in isolation; but need so we can test if
\r
1200 something was matched. Fixed.
\r
1202 * Lots of little bugs fixed in $x.y, %... translation due to new
\r
1203 action translator.
\r
1205 * Improperly tracking block nesting level; result was that you couldn't
\r
1206 see $ID in action of rule "a : A+ | ID {Token t = $ID;} | C ;"
\r
1208 * a : ID ID {$ID.text;} ; did not get a warning about ambiguous $ID ref.
\r
1210 * No error was found on $COMMENT.text:
\r
1213 : '/*' (options {greedy=false;} : . )* '*/'
\r
1214 {System.out.println("found method "+$COMMENT.text);}
\r
1217 $enclosinglexerrule scope does not exist. Use text or setText() here.
\r
1221 * Single return values are initialized now to default or to your spec.
\r
1223 * cleaned up input stream stuff. Added ANTLRReaderStream, ANTLRInputStream
\r
1224 and refactored. You can specify encodings now on ANTLRFileStream (and
\r
1225 ANTLRInputStream) now.
\r
1227 * You can set text local var now in a lexer rule and token gets that text.
\r
1228 start/stop indexes are still set for the token.
\r
1230 * Changed lexer slightly. Calling a nonfragment rule from a
\r
1231 nonfragment rule does not set the overall token.
\r
1235 * Fixed bug where unnecessary escapes yield char==0 like '\{'.
\r
1237 * Fixed analysis bug. This grammar didn't report a recursion warning:
\r
1244 The DFAState.equals() method was messed up.
\r
1246 * Added @synpredgate {...} action so you can tell ANTLR how to gate actions
\r
1247 in/out during syntactic predicate evaluation.
\r
1249 * Fuzzy parsing should be more efficient. It should backtrack over a rule
\r
1250 and then rewind and do it again "with feeling" to exec actions. It was
\r
1251 actually doing it 3x not 2x.
\r
1255 * Gutted and rebuilt the action translator for $x.y, $x::y, ...
\r
1256 Uses ANTLR v3 now for the first time inside v3 source. :)
\r
1257 ActionTranslator.java
\r
1259 * Fixed a bug where referencing a return value on a rule didn't work
\r
1260 because later a ref to that rule's predefined properties didn't
\r
1261 properly force a return value struct to be built. Added unit test.
\r
1265 * New DFA mechanisms. Cyclic DFA are implemented as state tables,
\r
1266 encoded via strings as java cannot handle large static arrays :(
\r
1267 States with edges emanating that have predicates are specially
\r
1268 treated. A method is generated to do these states. The DFA
\r
1269 simulation routine uses the "special" array to figure out if the
\r
1270 state is special. See March 25, 2006 entry for description:
\r
1271 http://www.antlr.org/blog/antlr3/codegen.tml. analysis.DFA now has
\r
1272 all the state tables generated for code gen. CyclicCodeGenerator.java
\r
1273 disappeared as it's unneeded code. :)
\r
1275 * Internal general clean up of the DFA.states vs uniqueStates thing.
\r
1276 Fixed lookahead decisions no longer fill uniqueStates. Waste of
\r
1277 time. Also noted that when adding sem pred edges, I didn't check
\r
1278 for state reuse. Fixed.
\r
1282 * When resolving ambig DFA states predicates, I did not add the new states
\r
1283 to the list of unique DFA states. No observable effect on output except
\r
1284 that DFA state numbers were not always contiguous for predicated decisions.
\r
1285 I needed this fix for new DFA tables.
\r
1287 3.0ea10 - June 2, 2006
\r
1291 * Improved grammar stats and added syntactic pred tracking.
\r
1295 * Due to a type mismatch, the DebugParser.recoverFromMismatchedToken()
\r
1296 method was not called. Debug events for mismatched token error
\r
1297 notification were not sent to ANTLRWorks probably
\r
1299 * Added getBacktrackingLevel() for any recognizer; needed for profiler.
\r
1301 * Only writes profiling data for antlr grammar analysis with -profile set
\r
1303 * Major update and bug fix to (runtime) Profiler.
\r
1307 * Added Lexer.skip() to force lexer to ignore current token and look for
\r
1308 another; no token is created for current rule and is not passed on to
\r
1309 parser (or other consumer of the lexer).
\r
1311 * Parsers are much faster now. I removed use of java.util.Stack for pushing
\r
1312 follow sets and use a hardcoded array stack instead. Dropped from
\r
1313 5900ms to 3900ms for parse+lex time parsing entire java 1.4.2 source. Lex
\r
1314 time alone was about 1500ms. Just looking at parse time, we get about 2x
\r
1315 speed improvement. :)
\r
1319 * Fixed NFA construction so it generates NFA for (A*)* such that ANTLRWorks
\r
1320 can display it properly.
\r
1324 * added abort method to Grammar so AW can terminate the conversion if it's
\r
1329 * added method to get left recursive rules from grammar without doing full
\r
1332 * analysis, code gen not attempted if serious error (like
\r
1333 left-recursion or missing rule definition) occurred while reading
\r
1334 the grammar in and defining symbols.
\r
1336 * added amazing optimization; reduces analysis time by 90% for java
\r
1337 grammar; simple IF statement addition!
\r
1339 3.0ea9 - May 20, 2006
\r
1341 * added global k value for grammar to limit lookahead for all decisions unless
\r
1342 overridden in a particular decision.
\r
1344 * added failsafe so that any decision taking longer than 2 seconds to create
\r
1345 the DFA will fall back on k=1. Use -ImaxtimeforDFA n (in ms) to set the time.
\r
1347 * added an option (turned off for now) to use multiple threads to
\r
1348 perform grammar analysis. Not much help on a 2-CPU computer as
\r
1349 garbage collection seems to peg the 2nd CPU already. :( Gotta wait for
\r
1352 * switched from #src to // $ANTLR src directive.
\r
1354 * CommonTokenStream.getTokens() looked past end of buffer sometimes. fixed.
\r
1356 * unicode literals didn't really work in DOT output and generated code. fixed.
\r
1358 * fixed the unit test rig so it compiles nicely with Java 1.5
\r
1360 * Added ant build.xml file (reads build.properties file)
\r
1362 * predicates sometimes failed to compile/eval properly due to missing (...)
\r
1363 in IF expressions. Forced (..)
\r
1365 * (...)? with only one alt were not optimized. Was:
\r
1367 // t.g:4:7: ( B )?
\r
1369 int LA1_0 = input.LA(1);
\r
1373 else if ( LA1_0==-1 ) {
\r
1377 NoViableAltException nvae =
\r
1378 new NoViableAltException("4:7: ( B )?", 1, 0, input);
\r
1384 // t.g:4:7: ( B )?
\r
1386 int LA1_0 = input.LA(1);
\r
1391 Smaller, faster and more readable.
\r
1393 * Allow manual init of return values now:
\r
1394 functionHeader returns [int x=3*4, char (*f)()=null] : ... ;
\r
1396 * Added optimization for DFAs that fixed a codegen bug with rules in lexer:
\r
1398 ASSIGNOP : '=' | '+=' ;
\r
1399 EQ is a subset of other rule. It did not given an error which is
\r
1400 correct, but generated bad code.
\r
1402 * ANTLR was sending column not char position to ANTLRWorks.
\r
1404 * Bug fix: location 0, 0 emitted for synpreds and empty alts.
\r
1406 * debugging event handshake how sends grammar file name. Added getGrammarFileName() to recognizers. Java.stg generates it:
\r
1408 public String getGrammarFileName() { return "<fileName>"; }
\r
1410 * tree parsers can do arbitrary lookahead now including backtracking. I
\r
1411 updated CommonTreeNodeStream.
\r
1413 * added events for debugging tree parsers:
\r
1415 /** Input for a tree parser is an AST, but we know nothing for sure
\r
1416 * about a node except its type and text (obtained from the adaptor).
\r
1417 * This is the analog of the consumeToken method. Again, the ID is
\r
1418 * the hashCode usually of the node so it only works if hashCode is
\r
1419 * not implemented.
\r
1421 public void consumeNode(int ID, String text, int type);
\r
1423 /** The tree parser looked ahead */
\r
1424 public void LT(int i, int ID, String text, int type);
\r
1426 /** The tree parser has popped back up from the child list to the
\r
1429 public void goUp();
\r
1431 /** The tree parser has descended to the first child of a the current
\r
1434 public void goDown();
\r
1436 * Added DebugTreeNodeStream and DebugTreeParser classes
\r
1438 * Added ctor because the debug tree node stream will need to ask quesitons about nodes and since nodes are just Object, it needs an adaptor to decode the nodes and get text/type info for the debugger.
\r
1440 public CommonTreeNodeStream(TreeAdaptor adaptor, Tree tree);
\r
1442 * added getter to TreeNodeStream:
\r
1443 public TreeAdaptor getTreeAdaptor();
\r
1445 * Implemented getText/getType in CommonTreeAdaptor.
\r
1447 * Added TraceDebugEventListener that can dump all events to stdout.
\r
1449 * I broke down and make Tree implement getText
\r
1451 * tree rewrites now gen location debug events.
\r
1453 * added AST debug events to listener; added blank listener for convenience
\r
1455 * updated debug events to send begin/end backtrack events for debugging
\r
1457 * with a : (b->b) ('+' b -> ^(PLUS $a b))* ; you get b[0] each time as
\r
1458 there is no loop in rewrite rule itself. Need to know context that
\r
1459 the -> is inside the rule and hence b means last value of b not all
\r
1462 * Bug in TokenRewriteStream; ops at indexes < start index blocked proper op.
\r
1464 * Actions in ST rewrites "-> ({$op})()" were not translated
\r
1466 * Added new action name:
\r
1469 catch (RecognitionException re) {
\r
1471 recover(input,re);
\r
1473 catch (Throwable t) {
\r
1474 System.err.println(t);
\r
1477 Overrides rule catch stuff.
\r
1479 * Isolated $ refs caused exception
\r
1481 3.0ea8 - March 11, 2006
\r
1483 * added @finally {...} action like @init for rules. Executes in
\r
1484 finally block (java target) after all other stuff like rule memoization.
\r
1485 No code changes needs; ST just refs a new action:
\r
1486 <ruleDescriptor.actions.finally>
\r
1488 * hideous bug fixed: PLUS='+' didn't result in '+' rule in lexer
\r
1490 * TokenRewriteStream didn't do toString() right when no rewrites had been done.
\r
1492 * lexer errors in interpreter were not printed properly
\r
1494 * bitsets are dumped in hex not decimal now for FOLLOW sets
\r
1496 * /* epsilon */ is not printed now when printing out grammars with empty alts
\r
1498 * Fixed another bug in tree rewrite stuff where it was checking that elements
\r
1499 had at least one element. Strange...commented out for now to see if I can remember what's up.
\r
1501 * Tree rewrites had problems when you didn't have x+=FOO variables. Rules
\r
1502 like this work now:
\r
1504 a : (x=ID)? y=ID -> ($x $y)?;
\r
1506 * filter=true for lexers turns on k=1 and backtracking for every token
\r
1507 alternative. Put the rules in priority order.
\r
1509 * added getLine() etc... to Tree to support better error reporting for
\r
1510 trees. Added MismatchedTreeNodeException.
\r
1512 * $templates::foo() is gone. added % as special template symbol.
\r
1513 %foo(a={},b={},...) ctor (even shorter than $templates::foo(...))
\r
1514 %({name-expr})(a={},...) indirect template ctor reference
\r
1516 The above are parsed by antlr.g and translated by codegen.g
\r
1517 The following are parsed manually here:
\r
1519 %{string-expr} anonymous template from string expr
\r
1520 %{expr}.y = z; template attribute y of StringTemplate-typed expr to z
\r
1521 %x.y = z; set template attribute y of x (always set never get attr)
\r
1522 to z [languages like python without ';' must still use the
\r
1523 ';' which the code generator is free to remove during code gen]
\r
1525 * -> ({expr})(a={},...) notation for indirect template rewrite.
\r
1526 expr is the name of the template.
\r
1528 * $x[i]::y and $x[-i]::y notation for accesssing absolute scope stack
\r
1529 indexes and relative negative scopes. $x[-1]::y is the y attribute
\r
1530 of the previous scope (stack top - 1).
\r
1532 * filter=true mode for lexers; can do this now...upon mismatch, just
\r
1533 consumes a char and tries again:
\r
1534 lexer grammar FuzzyJava;
\r
1535 options {filter=true;}
\r
1538 : TYPE WS? name=ID WS? (';'|'=')
\r
1539 {System.out.println("found var "+$name.text);}
\r
1542 * refactored char streams so ANTLRFileStream is now a subclass of
\r
1543 ANTLRStringStream.
\r
1545 * char streams for lexer now allowed nested backtracking in lexer.
\r
1547 * added TokenLabelType for lexer/parser for all token labels
\r
1549 * line numbers for error messages were not updated properly in antlr.g
\r
1550 for strings, char literals and <<...>>
\r
1552 * init action in lexer rules was before the type,start,line,... decls.
\r
1554 * Tree grammars can now specify output; I've only tested output=templat
\r
1557 * You can reference EOF now in the parser and lexer. It's just token type
\r
1560 * Bug fix: $ID refs in the *lexer* were all messed up. Cleaned up the
\r
1561 set of properties available...
\r
1563 * Bug fix: .st not found in rule ref when rule has scope:
\r
1566 StringTemplate funcDef;
\r
1569 {$field::funcDef = $field.st;}
\r
1571 it gets field_stack.st instead
\r
1573 * return in backtracking must return retval or null if return value.
\r
1575 * $property within a rule now works like $text, $st, ...
\r
1577 * AST/Template Rewrites were not gated by backtracking==0 so they
\r
1578 executed even when guessing. Auto AST construction is now gated also.
\r
1580 * CommonTokenStream was somehow returning tokens not text in toString()
\r
1582 * added useful methods to runtime.BitSet and also to CommonToken so you can
\r
1583 update the text. Added nice Token stream method:
\r
1585 /** Given a start and stop index, return a List of all tokens in
\r
1586 * the token type BitSet. Return null if no tokens were found. This
\r
1587 * method looks at both on and off channel tokens.
\r
1589 public List getTokens(int start, int stop, BitSet types);
\r
1591 * literals are now passed in the .tokens files so you can ref them in
\r
1592 tree parses, for example.
\r
1594 * added basic exception handling; no labels, just general catches:
\r
1598 catch[RecognitionException re] {
\r
1599 System.out.println("recog error");
\r
1601 catch[Exception e] {
\r
1602 System.out.println("error");
\r
1605 * Added method to TokenStream:
\r
1606 public String toString(Token start, Token stop);
\r
1608 * antlr generates #src lines in lexer grammars generated from combined grammars
\r
1609 so error messages refer to original file.
\r
1611 * lexers generated from combined grammars now use originally formatting.
\r
1613 * predicates have $x.y stuff translated now. Warning: predicates might be
\r
1614 hoisted out of context.
\r
1616 * return values in return val structs are now public.
\r
1618 * output=template with return values on rules was broken. I assume return values with ASTs was broken too. Fixed.
\r
1620 3.0ea7 - December 14, 2005
\r
1622 * Added -print option to print out grammar w/o actions
\r
1624 * Renamed BaseParser to be BaseRecognizer and even made Lexer derive from
\r
1625 this; nice as it now shares backtracking support code.
\r
1627 * Added syntactic predicates (...)=>. See December 4, 2005 entry:
\r
1629 http://www.antlr.org/blog/antlr3/lookahead.tml
\r
1631 Note that we have a new option for turning off rule memoization during
\r
1634 -nomemo when backtracking don't generate memoization code
\r
1636 * Predicates are now tested in order that you specify the alts. If you
\r
1637 leave the last alt "naked" (w/o pred), it will assume a true pred rather
\r
1638 than union of other preds.
\r
1640 * Added gated predicates "{p}?=>" that literally turn off a production whereas
\r
1641 disambiguating predicates are only hoisted into the predictor when syntax alone
\r
1642 is not sufficient to uniquely predict alternatives.
\r
1645 B : {!p}? => ("a"|"b")+ ;
\r
1647 * bug fixed related to predicates in predictor
\r
1650 B : {!p}? ("a"|"b")+ ;
\r
1651 DFA is correct. A state splits for input "a" on the pred.
\r
1652 Generated code though was hosed. No pred tests in prediction code!
\r
1653 I added testLexerPreds() and others in TestSemanticPredicateEvaluation.java
\r
1655 * added execAction template in case we want to do something in front of
\r
1656 each action execution or something.
\r
1658 * left-recursive cycles from rules w/o decisions were not detected.
\r
1660 * undefined lexer rules were not announced! fixed.
\r
1662 * unreachable messages for Tokens rule now indicate rule name not alt. E.g.,
\r
1664 Ruby.lexer.g:24:1: The following token definitions are unreachable: IVAR
\r
1666 * nondeterminism warnings improved for Tokens rule:
\r
1668 Ruby.lexer.g:10:1: Multiple token rules can match input such as ""0".."9"": INT, FLOAT
\r
1669 As a result, tokens(s) FLOAT were disabled for that input
\r
1672 * DOT diagrams didn't show escaped char properly.
\r
1674 * Char/string literals are now all 'abc' not "abc".
\r
1676 * action syntax changed "@scope::actionname {action}" where scope defaults
\r
1677 to "parser" if parser grammar or combined grammar, "lexer" if lexer grammar,
\r
1678 and "treeparser" if tree grammar. The code generation targets decide
\r
1679 what scopes are available. Each "scope" yields a hashtable for use in
\r
1680 the output templates. The scopes full of actions are sent to all output
\r
1681 file templates (currently headerFile and outputFile) as attribute actions.
\r
1682 Then you can reference <actions.scope> to get the map of actions associated
\r
1683 with scope and <actions.parser.header> to get the parser's header action
\r
1684 for example. This should be very flexible. The target should only have
\r
1685 to define which scopes are valid, but the action names should be variable
\r
1686 so we don't have to recompile ANTLR to add actions to code gen templates.
\r
1689 options {language=Java;}
\r
1690 @header { package foo; }
\r
1691 @parser::stuff { int i; } // names within scope not checked; target dependent
\r
1692 @members { int i; }
\r
1693 @lexer::header {head}
\r
1694 @lexer::members { int j; }
\r
1695 @headerfile::blort {...} // error: this target doesn't have headerfile
\r
1696 @treeparser::members {...} // error: this is not a tree parser
\r
1703 For now, the Java target uses members and header as a valid name. Within a
\r
1704 rule, the init action name is valid.
\r
1706 * changed $dynamicscope.value to $dynamicscope::value even if value is defined
\r
1707 in same rule such as $function::name where rule function defines name.
\r
1709 * $dynamicscope gets you the stack
\r
1711 * rule scopes go like this now:
\r
1715 scope slist,Symbols;
\r
1719 * Created RuleReturnScope as a generic rule return value. Makes it easier
\r
1721 RuleReturnScope r = parser.program();
\r
1722 System.out.println(r.getTemplate().toString());
\r
1724 * $template, $tree, $start, etc...
\r
1726 * $r.x in current rule. $r is ignored as fully-qualified name. $r.start works too
\r
1728 * added warning about $r referring to both return value of rule and dynamic scope of rule
\r
1730 * integrated StringTemplate in a very simple manner
\r
1733 -> template(arglist) "..."
\r
1734 -> template(arglist) <<...>>
\r
1735 -> namedTemplate(arglist)
\r
1736 -> {free expression}
\r
1740 a : A B -> {p1}? foo(a={$A.text})
\r
1741 -> {p2}? foo(a={$B.text})
\r
1742 -> // return nothing
\r
1744 An arg list is just a list of template attribute assignments to actions in curlies.
\r
1746 There is a setTemplateLib() method for you to use with named template rewrites.
\r
1751 options {output=template;}
\r
1754 This all should work for tree grammars too, but I'm still testing.
\r
1756 * fixed bugs where strings were improperly escaped in exceptions, comments, etc.. For example, newlines came out as newlines not the escaped version
\r
1758 3.0ea6 - November 13, 2005
\r
1760 * turned off -debug/-profile, which was on by default
\r
1762 * completely refactored the output templates; added some missing templates.
\r
1764 * dramatically improved infinite recursion error messages (actually
\r
1765 left-recursion never even was printed out before).
\r
1767 * wasn't printing dangling state messages when it reanalyzes with k=1.
\r
1769 * fixed a nasty bug in the analysis engine dealing with infinite recursion.
\r
1770 Spent all day thinking about it and cleaned up the code dramatically.
\r
1771 Bug fixed and software is more powerful and I understand it better! :)
\r
1773 * improved verbose DFA nodes; organized by alt
\r
1775 * got much better random phrase generation. For example:
\r
1777 $ java org.antlr.tool.RandomPhrase simple.g program
\r
1778 int Ktcdn ';' method wh '(' ')' '{' return 5 ';' '}'
\r
1780 * empty rules like "a : ;" generated code that didn't compile due to
\r
1781 try/catch for RecognitionException. Generated code couldn't possibly
\r
1782 throw that exception.
\r
1784 * when printing out a grammar, such as in comments in generated code,
\r
1785 ANTLR didn't print ast suffix stuff back out for literals.
\r
1787 * This never exited loop:
\r
1788 DATA : (options {greedy=false;}: .* '\n' )* '\n' '.' ;
\r
1789 and now it works due to new default nongreedy .* Also this works:
\r
1790 DATA : (options {greedy=false;}: .* '\n' )* '.' ;
\r
1792 * Dot star ".*" syntax didn't work; in lexer it is nongreedy by
\r
1793 default. In parser it is on greedy but also k=1 by default. Added
\r
1794 unit tests. Added blog entry to describe.
\r
1796 * ~T where T is the only token yielded an empty set but no error
\r
1798 * Used to generate unreachable message here:
\r
1805 z.g:3:11: The following alternatives are unreachable: 2
\r
1807 In fact it should really be an error; now it generates:
\r
1809 no start rule in grammar t (no rule can obviously be followed by EOF)
\r
1811 Per next change item, ANTLR cannot know that EOF follows rule 'a'.
\r
1813 * added error message indicating that ANTLR can't figure out what your
\r
1814 start rule is. Required to properly generate code in some cases.
\r
1816 * validating semantic predicates now work (if they are false, they
\r
1817 throw a new FailedPredicateException
\r
1819 * two hideous bug fixes in the IntervalSet, which made analysis go wrong
\r
1820 in a few cases. Thanks to Oliver Zeigermann for finding lots of bugs
\r
1821 and making suggested fixes (including the next two items)!
\r
1823 * cyclic DFAs are now nonstatic and hence can access instance variables
\r
1825 * labels are now allowed on lexical elements (in the lexer)
\r
1827 * added some internal debugging options
\r
1829 * ~'a'* and ~('a')* were not working properly; refactored antlr.g grammar
\r
1831 3.0ea5 - July 5, 2005
\r
1833 * Using '\n' in a parser grammar resulted in a nonescaped version of '\n' in the token names table making compilation fail. I fixed this by reorganizing/cleaning up portion of ANTLR that deals with literals. See comment org.antlr.codegen.Target.
\r
1835 * Target.getMaxCharValue() did not use the appropriate max value constant.
\r
1837 * ALLCHAR was a constant when it should use the Target max value def. set complement for wildcard also didn't use the Target def. Generally cleaned up the max char value stuff.
\r
1839 * Code gen didn't deal with ASTLabelType properly...I think even the 3.0ea7 example tree parser was broken! :(
\r
1841 * Added a few more unit tests dealing with escaped literals
\r
1843 3.0ea4 - June 29, 2005
\r
1845 * tree parsers work; added CommonTreeNodeStream. See simplecTreeParser
\r
1846 example in examples-v3 tarball.
\r
1848 * added superClass and ASTLabelType options
\r
1850 * refactored Parser to have a BaseParser and added TreeParser
\r
1852 * bug fix: actions being dumped in description strings; compile errors
\r
1855 3.0ea3 - June 23, 2005
\r
1859 * Automatic tree construction operators are in: ! ^ ^^
\r
1861 * Tree construction rewrite rules are in
\r
1862 -> {pred1}? rewrite1
\r
1863 -> {pred2}? rewrite2
\r
1867 The rewrite rules may be elements like ID, expr, $label, {node expr}
\r
1868 and trees ^( <root> <children> ). You have have (...)?, (...)*, (...)+
\r
1871 You may have rewrites in subrules not just at outer level of rule, but
\r
1872 any -> rewrite forces auto AST construction off for that alternative
\r
1875 To avoid cycles, copy semantics are used:
\r
1877 r : INT -> INT INT ;
\r
1879 means make two new nodes from the same INT token.
\r
1881 Repeated references to a rule element implies a copy for at least one
\r
1884 a : atom -> ^(atom atom) ; // NOT CYCLE! (dup atom tree)
\r
1886 * $ruleLabel.tree refers to tree created by matching the labeled element.
\r
1888 * A description of the blocks/alts is generated as a comment in output code
\r
1890 * A timestamp / signature is put at top of each generated code file
\r
1892 3.0ea2 - June 12, 2005
\r
1896 * Some error messages were missing the stackTrace parameter
\r
1898 * Removed the file locking mechanism as it's not cross platform
\r
1900 * Some absolute vs relative path name problems with writing output
\r
1901 files. Rules are now more concrete. -o option takes precedence
\r
1902 // -o /tmp /var/lib/t.g => /tmp/T.java
\r
1903 // -o subdir/output /usr/lib/t.g => subdir/output/T.java
\r
1904 // -o . /usr/lib/t.g => ./T.java
\r
1905 // -o /tmp subdir/t.g => /tmp/subdir/t.g
\r
1906 // If they didn't specify a -o dir so just write to location
\r
1907 // where grammar is, absolute or relative
\r
1909 * does error checking on unknown option names now
\r
1911 * Using just language code not locale name for error message file. I.e.,
\r
1912 the default (and for any English speaking locale) is en.stg not en_US.stg
\r
1915 * The error manager now asks the Tool to panic rather than simply doing
\r
1918 * Lots of refactoring concerning grammar, rule, subrule options. Now
\r
1919 detects invalid options.
\r
1921 3.0ea1 - June 1, 2005
\r
1923 Initial early access release
\r