What is token , lexeme and pattern?
Token
A token is a pain consisting of a token name and an optional attribute value. The token name is an abstract symbol representing a kind of lexical unit.A particular denoting an identifier.The token names are the input symbol that are the parser process.
Typically tokens are ,
1) Identifiers
2) Keywords
3) Operator
4) special symbol 5)constant
Pattern
A pattern is a description of the form that the lexeme’s of a token may take. In the case of keyword as a token , the pattern is just the sequence of characters that form the keyword. For identifiers and some other tokens. The pattern is more complex structure.That is matched by many string.
Lexeme
A lexeme is a sequence of character in the source program that matches pattern for a token and identified by the lexical analyzer as an instance of that token.
Example : while( y > = t ) = y -3
Will be represented by the set of pairs.
