Create Token Structure -- C Implementation

[codingwithbatista]

New Library and New Structure

Create a new library to new lexical structures and functions.

This library 'll contain some kind of structures, some o these are:

• Token
• Generic List (to support token lists)

Token Structure

A token's composed by the following features:

1. lexeme (it represents a word as it's written)
2. value (if the token is a stopword or math symbol, for example)
3. line number (the number of line that token occurs)
4. initial character column number (to count the characters in any word, in later analysis)
5. final character column number (to count the character where a token has been finished)