Add comments to the Lexer module

[LunaStev]

In the core functions of the front/lexer/src/lexer/lexer.rs file, please add clear comments explaining what each function does. The goal is to make the code more approachable, especially for first-time contributors.

Details:

• Focus on adding concise but descriptive comments above each core function.
• Use plain English so that even someone unfamiliar with the codebase can follow along.
• Avoid over-explaining trivial syntax, but clarify the logic or purpose of the function.

Why this is needed:

This will improve code readability and help newcomers understand the lexer module more easily, making it a good first issue for contributors.