语法错误地抓住了一个错误的规则，虽然在那个规则中没有 ' 而在我的文本中我有 '

Question

我有如下语法

grammar STIXPattern;

pattern
  : observationExpressions EOF
  ;

observationExpressions
  : <assoc=left> observationExpressions FOLLOWEDBY observationExpressions #observationExpressionsFollowedBY
  | observationExpressionOr                                               #observationExpressionOr_
  ;

observationExpressionOr
  : <assoc=left> observationExpressionOr OR observationExpressionOr     #observationExpressionOred
  | observationExpressionAnd                                            #observationExpressionAnd_
  ;

observationExpressionAnd
  : <assoc=left> observationExpressionAnd AND observationExpressionAnd  #observationExpressionAnded
  | observationExpression                                               #observationExpression_
  ;

observationExpression
  : LBRACK comparisonExpression RBRACK        # observationExpressionSimple
  | LPAREN observationExpressions RPAREN      # observationExpressionCompound
  | observationExpression startStopQualifier  # observationExpressionStartStop
  | observationExpression withinQualifier     # observationExpressionWithin
  | observationExpression repeatedQualifier   # observationExpressionRepeated
  ;

comparisonExpression
  : <assoc=left> comparisonExpression OR comparisonExpression         #comparisonExpressionOred
  | comparisonExpressionAnd                                           #comparisonExpressionAnd_
  ;

comparisonExpressionAnd
  : <assoc=left> comparisonExpressionAnd AND comparisonExpressionAnd  #comparisonExpressionAnded
  | propTest                                                          #comparisonExpressionAndpropTest
  ;

propTest
  : objectPath NOT? (EQ|NEQ) primitiveLiteral       # propTestEqual
  | objectPath NOT? (GT|LT|GE|LE) orderableLiteral  # propTestOrder
  | objectPath NOT? IN setLiteral                   # propTestSet
  | objectPath NOT? LIKE StringLiteral              # propTestLike
  | objectPath NOT? MATCHES StringLiteral           # propTestRegex
  | objectPath NOT? ISSUBSET StringLiteral          # propTestIsSubset
  | objectPath NOT? ISSUPERSET StringLiteral        # propTestIsSuperset
  | LPAREN comparisonExpression RPAREN              # propTestParen
  ;


startStopQualifier
  : START TimestampLiteral STOP TimestampLiteral
  ;

withinQualifier
  : WITHIN (IntPosLiteral|FloatPosLiteral) SECONDS
  ;

repeatedQualifier
  : REPEATS IntPosLiteral TIMES
  ;

objectPath
  : objectType COLON firstPathComponent objectPathComponent?
  ;

objectType
  : IdentifierWithoutHyphen
  | IdentifierWithHyphen
  ;

objectPathThl
  : varThlType DOT firstPathComponent objectPathComponent?
  ;

varThlType
  : IdentifierWithoutHyphen
  | IdentifierWithHyphen
  ;

firstPathComponent
  : IdentifierWithoutHyphen
  | StringLiteral
  ;

objectPathComponent
  : <assoc=left> objectPathComponent objectPathComponent  # pathStep
  | '.' (IdentifierWithoutHyphen | StringLiteral)         # keyPathStep
  | LBRACK (IntPosLiteral|IntNegLiteral|ASTERISK) RBRACK  # indexPathStep
  ;

setLiteral
  : LPAREN RPAREN
  | LPAREN primitiveLiteral (COMMA primitiveLiteral)* RPAREN
  ;

primitiveLiteral
  : orderableLiteral
  | BoolLiteral
  | objectPathThl
  ;

orderableLiteral
  : IntPosLiteral
  | IntNegLiteral
  | FloatPosLiteral
  | FloatNegLiteral
  | StringLiteral
  | BinaryLiteral
  | HexLiteral
  | TimestampLiteral
  ;

IntNegLiteral :
  '-' ('0' | [1-9] [0-9]*)
  ;

IntPosLiteral :
  '+'? ('0' | [1-9] [0-9]*)
  ;

FloatNegLiteral :
  '-' [0-9]* '.' [0-9]+
  ;

FloatPosLiteral :
  '+'? [0-9]* '.' [0-9]+
  ;

HexLiteral :
  'h' QUOTE TwoHexDigits* QUOTE
  ;

BinaryLiteral :
  'b' QUOTE
  ( Base64Char Base64Char Base64Char Base64Char )*
  ( (Base64Char Base64Char Base64Char Base64Char )
  | (Base64Char Base64Char Base64Char ) '='
  | (Base64Char Base64Char ) '=='
  )
  QUOTE
  ;

StringLiteral :
  QUOTE ( ~['\] | '\\'' | '\\' )* QUOTE
  ;

BoolLiteral :
  TRUE | FALSE
  ;

TimestampLiteral :
  't' QUOTE
  [0-9] [0-9] [0-9] [0-9] HYPHEN
  ( ('0' [1-9]) | ('1' [012]) ) HYPHEN
  ( ('0' [1-9]) | ([12] [0-9]) | ('3' [01]) )
  'T'
  ( ([01] [0-9]) | ('2' [0-3]) ) COLON
  [0-5] [0-9] COLON
  ([0-5] [0-9] | '60')
  (DOT [0-9]+)?
  'Z'
  QUOTE
  ;

//////////////////////////////////////////////
// Keywords

AND:  'AND' ;
OR:  'OR' ;
NOT:  'NOT' ;
FOLLOWEDBY: 'FOLLOWEDBY';
LIKE:  'LIKE' ;
MATCHES:  'MATCHES' ;
ISSUPERSET:  'ISSUPERSET' ;
ISSUBSET: 'ISSUBSET' ;
LAST:  'LAST' ;
IN:  'IN' ;
START:  'START' ;
STOP:  'STOP' ;
SECONDS:  'SECONDS' ;
TRUE:  'true' ;
FALSE:  'false' ;
WITHIN:  'WITHIN' ;
REPEATS:  'REPEATS' ;
TIMES:  'TIMES' ;

// After keywords, so the lexer doesn't tokenize them as identifiers.
// Object types may have unquoted hyphens, but property names
// (in object paths) cannot.
IdentifierWithoutHyphen :
  [a-zA-Z_] [a-zA-Z0-9_]*
  ;

IdentifierWithHyphen :
  [a-zA-Z_] [a-zA-Z0-9_-]*
  ;

EQ        :   '=' | '==';
NEQ       :   '!=' | '<>';
LT        :   '<';
LE        :   '<=';
GT        :   '>';
GE        :   '>=';

QUOTE     : '\'';
COLON     : ':' ;
DOT       : '.' ;
COMMA     : ',' ;
RPAREN    : ')' ;
LPAREN    : '(' ;
RBRACK    : ']' ;
LBRACK    : '[' ;
PLUS      : '+' ;
HYPHEN    : MINUS ;
MINUS     : '-' ;
POWER_OP  : '^' ;
DIVIDE    : '/' ;
ASTERISK  : '*';

fragment HexDigit: [A-Fa-f0-9];
fragment TwoHexDigits: HexDigit HexDigit;
fragment Base64Char: [A-Za-z0-9+/];

// Whitespace and comments
//
WS  :  [ \t\r\n\u000B\u000C\u0085\u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u2028\u2029\u202f\u205f\u3000]+ -> skip
    ;

COMMENT
    :   '/*' .*? '*/' -> skip
    ;

LINE_COMMENT
    :   '//' ~[\r\n]* -> skip
    ;

// Catch-all to prevent lexer from silently eating unusable characters.
InvalidCharacter
    : .
    ;

所以在这里 IdentifierWithoutHyphen 和 IdentifierWithHyphen 你可以看到我没有包括任何 ' 所以我假设如果我输入像 [file:created = t'2019-04-25T1:23:00.000Z'] 这样的文本我应该仍然看到 orderableLiteral 被触发，但令人惊讶的是我看到这个 objectPathThl 在 primitiveLiteral 中被触发。不知道为什么？我没有在 [a-zA-Z_] [a-zA-Z0-9_]* 或 [a-zA-Z_] [a-zA-Z0-9_-]* 中包含任何 ' 有什么想法吗？

Answer 1

如果您将词法分析器创建的标记打印到您的控制台（这始终是您应该做的第一件事！），您可以这样做：

String source = "[file:created = t'2019-04-25T1:23:00.000Z']";
STIXPatternLexer lexer = new STIXPatternLexer(CharStreams.fromString(source));
CommonTokenStream stream = new CommonTokenStream(lexer);
stream.fill();

for (Token t : stream.getTokens()) {
  System.out.printf("%-25s `%s`%n",
      STIXPatternLexer.VOCABULARY.getSymbolicName(t.getType()),
      t.getText().replace("\n", "\n"));
}

您将看到以下内容：

LBRACK                    `[`
IdentifierWithoutHyphen   `file`
COLON                     `:`
IdentifierWithoutHyphen   `created`
EQ                        `=`
IdentifierWithoutHyphen   `t`
StringLiteral             `'2019-04-25T1:23:00.000Z'`
RBRACK                    `]`
EOF                       `<EOF>`

如您所见，t'2019-04-25T1:23:00.000Z' 是 而不是 被标记为单个 TimestampLiteral（我怀疑您期望的），但作为 IdentifierWithoutHyphen 后跟 StringLiteral.

如果仔细看你的TimestampLiteral：

TimestampLiteral :
  't' QUOTE
  [0-9] [0-9] [0-9] [0-9] HYPHEN
  ( ('0' [1-9]) | ('1' [012]) ) HYPHEN
  ( ('0' [1-9]) | ([12] [0-9]) | ('3' [01]) )
  'T'
  ( ([01] [0-9]) | ('2' [0-3]) ) COLON
  [0-5] [0-9] COLON
  ([0-5] [0-9] | '60')
  (DOT [0-9]+)?
  'Z'
  QUOTE
  ;

然后在您的输入 t'2019-04-25T1:23:00.000Z' 中，您将看到输入由一位数小时组成，规则 TimestampLiteral 不匹配。

语法错误地抓住了一个错误的规则，虽然在那个规则中没有 ' 而在我的文本中我有 '

grammar mistakenly grab a wrong rule though in that rule there is no ' and in my text I have '

antlr

antlr4