From 8b087a7ecb2cb7fba09f3f4c9ef2ecb9fe6d56cc Mon Sep 17 00:00:00 2001 From: Nagajaideep Date: Fri, 5 Sep 2025 23:15:29 +0530 Subject: [PATCH 1/8] docs: Implement transpiler documentation improvements --- README.md | 21 ++ docs/index.md | 208 +++++++++++++---- .../astx_transpiler_refactor_tutorial.md | 218 ++++++++++++++++++ libs/astx-transpilers/README.md | 26 +++ 4 files changed, 423 insertions(+), 50 deletions(-) create mode 100644 docs/tutorials/astx_transpiler_refactor_tutorial.md diff --git a/README.md b/README.md index 55c1d47e..36fa7427 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,27 @@ example: >>> print(add_function.to_yaml()) ``` +## Transpilers + +ASTx includes a powerful transpiler system to convert your AST structures into executable Python code. This is great for code generation, prototyping, or building custom language tools. + +```python +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Using the 'add_function' ASTx node from the example above +transpiler = ASTxPythonTranspiler() +python_code = transpiler.visit(add_function) +print(python_code) +``` + +**Output:** +```python +def add(x: int, y: int) -> int: + return (x + y) +``` + +For a deep dive into the architecture and more hands-on examples, check out our **[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. + --- ## 📚 Documentation diff --git a/docs/index.md b/docs/index.md index 222932da..36fa7427 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,69 +1,177 @@ -# ASTx Library: A Versatile Toolkit for Language Representation +# ASTx: Abstract Syntax Tree Framework -ASTx is a groundbreaking library designed to encapsulate language components in -an agnostic and pythonic way. It provides a comprehensive set of classes and -functionalities, allowing developers to articulate the core elements of any -programming language. +![CI](https://img.shields.io/github/actions/workflow/status/arxlang/astx/main.yaml?logo=github&label=CI) +[![Python Versions](https://img.shields.io/pypi/pyversions/astx)](https://pypi.org/project/astx/) +[![Package Version](https://img.shields.io/pypi/v/astx?color=blue)](https://pypi.org/project/astx/) +![License](https://img.shields.io/pypi/l/astx?color=blue) +![Discord](https://img.shields.io/discord/966124290464428042?logo=discord&color=blue) -- License: BSD 3 Clause -- Documentation: https://astx.arxlang.org +ASTx is a versatile and extensible library for representing, manipulating, and +analyzing Abstract Syntax Trees (ASTs). It provides a unified interface for +working with ASTs in various contexts, such as compilers, interpreters, and +transpilers. -## Core Features +ASTx makes it easy to model programming languages, apply transformations, +generate code, and build custom tools for static and dynamic analysis. -### 1. **Expressive Language Components** +**ASTx** doesn't aim to be a `lexer` or a `parser`, although it could be used by +any programming language or parser in order to provide a high level +representation of the AST. -ASTx offers a rich suite of classes to describe essential language constructs -such as: +It integrates with [IRx](https://github.com/arxlang/irx), enabling code +generation with **LLVM**. Currently, only a small subset of **ASTx** nodes is +supported, but active development is underway, with full support expected soon. -- `If` statements -- `For` loops -- `Function` declarations and usages -- Variables -- Data Types -- Operations -- And more +**Note**: this project is under active development and it is not ready for +production yet. -These classes enable a concise and clear representation, providing an intuitive -way to model various programming constructs. +--- -### 2. **Symbol Table Class** +## 🚀 Features -An integral part of ASTx, the Symbol Table class facilitates the translation of -ASTx expressions to other languages like LLVM-IR. This class acts as a mapping -layer, allowing a seamless connection between ASTx expressions and target -language representations. +- **Language-Agnostic Design**: Model and manipulate ASTs for different + programming languages. +- **Extensibility**: Easily add support for new language features or custom AST + nodes. +- **Code Generation**: Transform ASTs into target code for various backends or + target languages. +- **Rich Node Set**: Support for common constructs like variables, expressions, + functions, classes, and control flow. +- **Python Integration**: Built with Python, making it easy to integrate into + Python-based projects. +- **Symbol Table**: Support for an initial implementation of Symbol Table. -### 3. **Language Agnostic Design** +--- -Uniquely tailored to be independent of specific programming languages, ASTx -offers a flexible foundation. It strives to provide initial components that can -describe any programming language, giving users the freedom to work with -multiple languages effortlessly. +## 📦 Installation -### 4. **Integration with Projects like ARX-IR** +Install ASTx from PyPI: -ASTx has proven to be a vital tool in projects like ARX-IR, where it's leveraged -to translate Abstract Syntax Trees (AST) into LLVM-IR. This showcases the -library's adaptability and potential to serve as a foundational layer in various -applications. +```bash +pip install astx +``` -## Why Choose ASTx? +--- -ASTx is not just a library; it's a robust framework that fosters creativity and -efficiency in language processing. Its pythonic design, combined with the power -to handle different language constructs, positions ASTx as an invaluable -resource for developers and researchers alike. +## 📖 Overview -Whether you're building a compiler, working on language translation, or -exploring new frontiers in programming language design, ASTx offers a reliable -and extensible toolkit to support your endeavors. +ASTx is designed around two primary concepts: -## Getting Started +1. **Nodes**: Each node represents a language construct (e.g., `Variable`, + `Function`, `IfStmt`). +2. **Tree**: Nodes are organized hierarchically, forming an abstract + representation of the program structure. -You can explore the ASTx library and dive into its capabilities by accessing the -official documentation. For those interested in contributing or seeking further -insights, the ASTx community provides extensive support and collaboration -opportunities. +Additionally, ASTx provides a simple transpiler for converting ASTx nodes to +Python code (in text format). This feature is intended solely for educational +purposes, demonstrating how a transpiler from ASTx to any other language can be +implemented. -Unlock the potential of language representation with ASTx, and join us in -shaping the future of programming languages. +--- + +## ✨ Usage + +### 1. Create an AST + +```python +import astx + +# Define a simple function `add(x, y): return x + y` +args = astx.Arguments( + astx.Argument(name="x", type_=astx.Int32()), + astx.Argument(name="y", type_=astx.Int32()), +) +fn_body = astx.Block() +fn_body.append( + astx.FunctionReturn( + value=astx.BinaryOp(op_code="+", lhs=astx.Variable("x"), rhs=astx.Variable("y")) + ) +) +add_function = astx.FunctionDef( + prototype=astx.FunctionPrototype(name="add", args=args, return_type=astx.Int32()), + body=fn_body, +) +``` + +### 2. Generate Code + +Use a transpiler to convert the AST to Python code: + +```python +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Transpile the AST to Python +transpiler = ASTxPythonTranspiler() +python_code = transpiler.visit(add_function) + +print(python_code) +``` + +Output: + +```python +def add(x: int, y: int) -> int: + return (x + y) +``` + +### 3. ASTx Visualization Features + +**ASTx** offers multiple ways to visualize the AST structure: + +- YAML +- JSON +- Graphical visualization (PNG or ASCII) + +In a Jupyter Notebook, the default graphical visualization is **PNG**, while in +a console, the default is **ASCII**. + +You can also print the AST structure in **JSON** or **YAML** format. For +example: + +```python +>>> print(add_function.to_json()) +>>> print(add_function.to_yaml()) +``` + +## Transpilers + +ASTx includes a powerful transpiler system to convert your AST structures into executable Python code. This is great for code generation, prototyping, or building custom language tools. + +```python +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Using the 'add_function' ASTx node from the example above +transpiler = ASTxPythonTranspiler() +python_code = transpiler.visit(add_function) +print(python_code) +``` + +**Output:** +```python +def add(x: int, y: int) -> int: + return (x + y) +``` + +For a deep dive into the architecture and more hands-on examples, check out our **[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. + +--- + +## 📚 Documentation + +Detailed documentation and examples can be found in the +[official documentation](https://arxlang.github.io/astx). + +--- + +## 🛠️ Contributing + +Contributions are welcome! Please check out our +[Contributing Guide](https://astx.arxlang.org/contributing/) for more +information. + +--- + +## 📝 License + +ASTx is open-source software licensed under the BSD-3-Clause License. See +[LICENSE](LICENSE) for details. diff --git a/docs/tutorials/astx_transpiler_refactor_tutorial.md b/docs/tutorials/astx_transpiler_refactor_tutorial.md new file mode 100644 index 00000000..7cf1309e --- /dev/null +++ b/docs/tutorials/astx_transpiler_refactor_tutorial.md @@ -0,0 +1,218 @@ +# ASTx Transpiler Tutorial + +This guide walks you through using ASTx's transpiler system to convert your AST structures into Python code. We've redesigned the transpiler with a cleaner architecture that's easier to use and maintain. + +## How the Transpiler Works + +The transpiler follows a straightforward two-step approach: + +### Step 1: ASTx → Python AST Objects + +The `ASTxPythonASTTranspiler` converts your ASTx nodes into Python's built-in AST objects. This is useful when you need to work with the AST programmatically. + +```python +from astx_transpilers.python_to_ast import ASTxPythonASTTranspiler + +# Get a Python AST object +ast_transpiler = ASTxPythonASTTranspiler() +python_ast = ast_transpiler.visit(your_astx_node) +``` + +### Step 2: ASTx → Python Source Code + +The `ASTxPythonTranspiler` takes your ASTx nodes and produces clean, readable Python source code. Behind the scenes, it uses the AST transpiler and then converts to string format. + +```python +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Get Python source code as a string +string_transpiler = ASTxPythonTranspiler() +python_code = string_transpiler.visit(your_astx_node) +``` + +## What's Better About This Approach + +- **No more circular imports**: Each component has a single, clear job +- **Easier debugging**: Problems in AST generation don't affect string formatting +- **More flexible**: Use AST objects directly or get formatted strings +- **Cleaner code**: Each module focuses on one thing and does it well + +## Testing Your Setup + +Make sure everything works by running the test suite: + +```bash +python -m pytest libs/astx-transpilers/tests/ -v +``` + +## Usage Examples + +### Example 1: Simple Integer Literal + +**Code:** +```python +import astx +from astx_transpilers.python_to_ast import ASTxPythonASTTranspiler +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Create an ASTx integer literal +astx_node = astx.LiteralInt32(value=42) + +# Convert to Python AST +python_ast = ASTxPythonASTTranspiler().visit(astx_node) +print(python_ast) + +# Convert to Python source code string +code_str = ASTxPythonTranspiler().visit(astx_node) +print(code_str) +``` + +**Output:** +``` +<_ast.Constant value=42> +42 +``` + +--- + +### Example 2: Lambda Expression + +**Code:** +```python +import astx +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Create lambda: lambda x: x + 1 +params = astx.Arguments(astx.Argument(name="x", type_=astx.Int32())) +body = astx.BinaryOp(op_code="+", lhs=astx.Variable(name="x"), rhs=astx.LiteralInt32(1)) +lambda_node = astx.LambdaExpr(params=params, body=body) + +# Convert to Python source code +transpiler = ASTxPythonTranspiler() +code_str = transpiler.visit(lambda_node) +print(code_str) +``` + +**Output:** +```python +lambda x: x + 1 +``` + +--- + +### Example 3: Mathematical Expression + +**Code:** +```python +import astx +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Create: (5 + 3) * 2 +left_expr = astx.BinaryOp( + op_code="+", + lhs=astx.LiteralInt32(5), + rhs=astx.LiteralInt32(3) +) +math_expr = astx.BinaryOp( + op_code="*", + lhs=left_expr, + rhs=astx.LiteralInt32(2) +) + +transpiler = ASTxPythonTranspiler() +result = transpiler.visit(math_expr) +print(result) +``` + +**Output:** +``` +((5 + 3) * 2) +``` + +--- + +### Example 4: Function Definition + +**Code:** +```python +import astx +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Create function: def add(x, y): return x + y +args = astx.Arguments( + astx.Argument(name="x", type_=astx.Int32()), + astx.Argument(name="y", type_=astx.Int32()), +) +fn_body = astx.Block() +fn_body.append( + astx.FunctionReturn( + value=astx.BinaryOp( + op_code="+", + lhs=astx.Variable("x"), + rhs=astx.Variable("y") + ) + ) +) +add_function = astx.FunctionDef( + prototype=astx.FunctionPrototype( + name="add", + args=args, + return_type=astx.Int32() + ), + body=fn_body, +) + +transpiler = ASTxPythonTranspiler() +code_str = transpiler.visit(add_function) +print(code_str) +``` + +**Output:** +```python +def add(x: int, y: int) -> int: + return (x + y) +``` + +--- + +### Example 5: Import Statement + +**Code:** +```python +import astx +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Create: import os, sys +import_stmt = astx.ImportStmt( + names=[ + astx.AliasExpr(name="os"), + astx.AliasExpr(name="sys") + ] +) + +transpiler = ASTxPythonTranspiler() +code_str = transpiler.visit(import_stmt) +print(code_str) +``` + +**Output:** +```python +import os, sys +``` + +## What You Get +- **AST Output:** Valid Python AST objects that can be further processed +- **String Output:** Clean Python source code ready to execute or save to files + +## Implementation Details +The refactored transpiler provides better code organization and eliminates previous circular dependency issues. Each component now has a single, clear purpose. + +--- + +**Source Files:** +- `libs/astx-transpilers/src/astx_transpilers/python_to_ast.py` +- `libs/astx-transpilers/src/astx_transpilers/python_string.py` + +**Tests:** +- `libs/astx-transpilers/tests/test_python_to_ast.py` +- `libs/astx-transpilers/tests/test_python_string.py` diff --git a/libs/astx-transpilers/README.md b/libs/astx-transpilers/README.md index 55c1d47e..f495ffbb 100644 --- a/libs/astx-transpilers/README.md +++ b/libs/astx-transpilers/README.md @@ -133,6 +133,32 @@ example: >>> print(add_function.to_yaml()) ``` +## Transpilers + +ASTx includes transpilers that can convert your AST structures into Python code. +This makes it easy to generate executable Python from your abstract +representations. + +```python +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Using the same add_function from above +transpiler = ASTxPythonTranspiler() +python_code = transpiler.visit(add_function) +print(python_code) +``` + +**Output:** + +```python +def add(x: int, y: int) -> int: + return (x + y) +``` + +The transpiler handles various ASTx constructs including functions, variables, +expressions, and control flow. For comprehensive examples and detailed usage, +check out our [transpiler documentation](https://arxlang.github.io/astx). + --- ## 📚 Documentation From 18210bc1239a43f9a9344d9f0d6507206ac13c26 Mon Sep 17 00:00:00 2001 From: Nagajaideep Date: Sat, 6 Sep 2025 11:36:36 +0530 Subject: [PATCH 2/8] removed marketing type of test --- README.md | 8 +- docs/index.md | 101 ++++++++---------- .../astx_transpiler_refactor_tutorial.md | 32 +++++- libs/astx-transpilers/README.md | 13 ++- 4 files changed, 86 insertions(+), 68 deletions(-) diff --git a/README.md b/README.md index 36fa7427..458675c0 100644 --- a/README.md +++ b/README.md @@ -135,7 +135,9 @@ example: ## Transpilers -ASTx includes a powerful transpiler system to convert your AST structures into executable Python code. This is great for code generation, prototyping, or building custom language tools. +ASTx includes a powerful transpiler system to convert your AST structures into +executable Python code. This is great for code generation, prototyping, or +building custom language tools. ```python from astx_transpilers.python_string import ASTxPythonTranspiler @@ -147,12 +149,14 @@ print(python_code) ``` **Output:** + ```python def add(x: int, y: int) -> int: return (x + y) ``` -For a deep dive into the architecture and more hands-on examples, check out our **[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. +For a deep dive into the architecture and more hands-on examples, check out our +**[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. --- diff --git a/docs/index.md b/docs/index.md index 36fa7427..ff2ebe8a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,40 +6,32 @@ ![License](https://img.shields.io/pypi/l/astx?color=blue) ![Discord](https://img.shields.io/discord/966124290464428042?logo=discord&color=blue) -ASTx is a versatile and extensible library for representing, manipulating, and -analyzing Abstract Syntax Trees (ASTs). It provides a unified interface for -working with ASTs in various contexts, such as compilers, interpreters, and -transpilers. +ASTx is a Python library for representing and working with Abstract Syntax Trees +(ASTs). It provides a unified interface for building tools such as compilers, +interpreters, and transpilers. -ASTx makes it easy to model programming languages, apply transformations, -generate code, and build custom tools for static and dynamic analysis. +With ASTx, you can define language constructs, transform ASTs, generate code, +and build tools for analysis. -**ASTx** doesn't aim to be a `lexer` or a `parser`, although it could be used by -any programming language or parser in order to provide a high level -representation of the AST. +ASTx is not a lexer or parser. It can be used together with parsers to provide a +higher-level representation of the AST. It integrates with [IRx](https://github.com/arxlang/irx), enabling code -generation with **LLVM**. Currently, only a small subset of **ASTx** nodes is -supported, but active development is underway, with full support expected soon. +generation with **LLVM**. Currently, only a subset of ASTx nodes is supported, +but active development is underway. -**Note**: this project is under active development and it is not ready for -production yet. +**Note:** This project is under active development and not ready for production +use. --- ## 🚀 Features -- **Language-Agnostic Design**: Model and manipulate ASTs for different - programming languages. -- **Extensibility**: Easily add support for new language features or custom AST - nodes. -- **Code Generation**: Transform ASTs into target code for various backends or - target languages. -- **Rich Node Set**: Support for common constructs like variables, expressions, - functions, classes, and control flow. -- **Python Integration**: Built with Python, making it easy to integrate into - Python-based projects. -- **Symbol Table**: Support for an initial implementation of Symbol Table. +- Language-agnostic: works across different programming languages +- Extensible: add new nodes or extend existing ones +- Code generation: convert ASTs to Python code (other backends planned) +- Includes common constructs: variables, functions, control flow, types +- Initial Symbol Table implementation included --- @@ -55,17 +47,16 @@ pip install astx ## 📖 Overview -ASTx is designed around two primary concepts: +ASTx is designed around two main concepts: -1. **Nodes**: Each node represents a language construct (e.g., `Variable`, - `Function`, `IfStmt`). -2. **Tree**: Nodes are organized hierarchically, forming an abstract - representation of the program structure. +1. **Nodes**: each node represents a language construct (e.g., `Variable`, + `Function`, `IfStmt`) +2. **Tree**: nodes are organized hierarchically, forming an abstract + representation of the program -Additionally, ASTx provides a simple transpiler for converting ASTx nodes to -Python code (in text format). This feature is intended solely for educational -purposes, demonstrating how a transpiler from ASTx to any other language can be -implemented. +ASTx also includes a simple transpiler for converting ASTx nodes to Python code +(text). This feature is for educational purposes, showing how a transpiler can +be implemented. --- @@ -76,7 +67,7 @@ implemented. ```python import astx -# Define a simple function `add(x, y): return x + y` +# Define a simple function: add(x, y) -> x + y args = astx.Arguments( astx.Argument(name="x", type_=astx.Int32()), astx.Argument(name="y", type_=astx.Int32()), @@ -100,7 +91,6 @@ Use a transpiler to convert the AST to Python code: ```python from astx_transpilers.python_string import ASTxPythonTranspiler -# Transpile the AST to Python transpiler = ASTxPythonTranspiler() python_code = transpiler.visit(add_function) @@ -114,64 +104,67 @@ def add(x: int, y: int) -> int: return (x + y) ``` -### 3. ASTx Visualization Features +### 3. Visualize ASTs -**ASTx** offers multiple ways to visualize the AST structure: +ASTx supports multiple visualization formats: - YAML - JSON -- Graphical visualization (PNG or ASCII) +- Graphical (PNG or ASCII) -In a Jupyter Notebook, the default graphical visualization is **PNG**, while in -a console, the default is **ASCII**. +In a Jupyter Notebook, the default visualization is **PNG**. In a console, the +default is **ASCII**. -You can also print the AST structure in **JSON** or **YAML** format. For -example: +Example: ```python ->>> print(add_function.to_json()) ->>> print(add_function.to_yaml()) +print(add_function.to_json()) +print(add_function.to_yaml()) ``` -## Transpilers +--- + +## 🔄 Transpilers -ASTx includes a powerful transpiler system to convert your AST structures into executable Python code. This is great for code generation, prototyping, or building custom language tools. +ASTx includes a transpiler system for converting AST structures into Python +code. This can be used for code generation, prototyping, or experimenting with +language tools. ```python from astx_transpilers.python_string import ASTxPythonTranspiler -# Using the 'add_function' ASTx node from the example above transpiler = ASTxPythonTranspiler() python_code = transpiler.visit(add_function) print(python_code) ``` **Output:** + ```python def add(x: int, y: int) -> int: return (x + y) ``` -For a deep dive into the architecture and more hands-on examples, check out our **[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. +For more examples, see the +**[transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. --- ## 📚 Documentation -Detailed documentation and examples can be found in the -[official documentation](https://arxlang.github.io/astx). +Full documentation and examples: +[https://arxlang.github.io/astx](https://arxlang.github.io/astx) --- ## 🛠️ Contributing -Contributions are welcome! Please check out our -[Contributing Guide](https://astx.arxlang.org/contributing/) for more -information. +Contributions are welcome! See the +[Contributing Guide](https://astx.arxlang.org/contributing/) for details. --- ## 📝 License -ASTx is open-source software licensed under the BSD-3-Clause License. See +ASTx is open-source software licensed under the **BSD-3-Clause License**. See [LICENSE](LICENSE) for details. diff --git a/docs/tutorials/astx_transpiler_refactor_tutorial.md b/docs/tutorials/astx_transpiler_refactor_tutorial.md index 7cf1309e..80b0c938 100644 --- a/docs/tutorials/astx_transpiler_refactor_tutorial.md +++ b/docs/tutorials/astx_transpiler_refactor_tutorial.md @@ -1,6 +1,8 @@ # ASTx Transpiler Tutorial -This guide walks you through using ASTx's transpiler system to convert your AST structures into Python code. We've redesigned the transpiler with a cleaner architecture that's easier to use and maintain. +This guide walks you through using ASTx's transpiler system to convert your AST +structures into Python code. We've redesigned the transpiler with a cleaner +architecture that's easier to use and maintain. ## How the Transpiler Works @@ -8,7 +10,8 @@ The transpiler follows a straightforward two-step approach: ### Step 1: ASTx → Python AST Objects -The `ASTxPythonASTTranspiler` converts your ASTx nodes into Python's built-in AST objects. This is useful when you need to work with the AST programmatically. +The `ASTxPythonASTTranspiler` converts your ASTx nodes into Python's built-in +AST objects. This is useful when you need to work with the AST programmatically. ```python from astx_transpilers.python_to_ast import ASTxPythonASTTranspiler @@ -20,7 +23,9 @@ python_ast = ast_transpiler.visit(your_astx_node) ### Step 2: ASTx → Python Source Code -The `ASTxPythonTranspiler` takes your ASTx nodes and produces clean, readable Python source code. Behind the scenes, it uses the AST transpiler and then converts to string format. +The `ASTxPythonTranspiler` takes your ASTx nodes and produces clean, readable +Python source code. Behind the scenes, it uses the AST transpiler and then +converts to string format. ```python from astx_transpilers.python_string import ASTxPythonTranspiler @@ -33,7 +38,8 @@ python_code = string_transpiler.visit(your_astx_node) ## What's Better About This Approach - **No more circular imports**: Each component has a single, clear job -- **Easier debugging**: Problems in AST generation don't affect string formatting +- **Easier debugging**: Problems in AST generation don't affect string + formatting - **More flexible**: Use AST objects directly or get formatted strings - **Cleaner code**: Each module focuses on one thing and does it well @@ -50,6 +56,7 @@ python -m pytest libs/astx-transpilers/tests/ -v ### Example 1: Simple Integer Literal **Code:** + ```python import astx from astx_transpilers.python_to_ast import ASTxPythonASTTranspiler @@ -68,6 +75,7 @@ print(code_str) ``` **Output:** + ``` <_ast.Constant value=42> 42 @@ -78,6 +86,7 @@ print(code_str) ### Example 2: Lambda Expression **Code:** + ```python import astx from astx_transpilers.python_string import ASTxPythonTranspiler @@ -94,6 +103,7 @@ print(code_str) ``` **Output:** + ```python lambda x: x + 1 ``` @@ -103,6 +113,7 @@ lambda x: x + 1 ### Example 3: Mathematical Expression **Code:** + ```python import astx from astx_transpilers.python_string import ASTxPythonTranspiler @@ -125,6 +136,7 @@ print(result) ``` **Output:** + ``` ((5 + 3) * 2) ``` @@ -134,6 +146,7 @@ print(result) ### Example 4: Function Definition **Code:** + ```python import astx from astx_transpilers.python_string import ASTxPythonTranspiler @@ -168,6 +181,7 @@ print(code_str) ``` **Output:** + ```python def add(x: int, y: int) -> int: return (x + y) @@ -178,6 +192,7 @@ def add(x: int, y: int) -> int: ### Example 5: Import Statement **Code:** + ```python import astx from astx_transpilers.python_string import ASTxPythonTranspiler @@ -196,23 +211,30 @@ print(code_str) ``` **Output:** + ```python import os, sys ``` ## What You Get + - **AST Output:** Valid Python AST objects that can be further processed - **String Output:** Clean Python source code ready to execute or save to files ## Implementation Details -The refactored transpiler provides better code organization and eliminates previous circular dependency issues. Each component now has a single, clear purpose. + +The refactored transpiler provides better code organization and eliminates +previous circular dependency issues. Each component now has a single, clear +purpose. --- **Source Files:** + - `libs/astx-transpilers/src/astx_transpilers/python_to_ast.py` - `libs/astx-transpilers/src/astx_transpilers/python_string.py` **Tests:** + - `libs/astx-transpilers/tests/test_python_to_ast.py` - `libs/astx-transpilers/tests/test_python_string.py` diff --git a/libs/astx-transpilers/README.md b/libs/astx-transpilers/README.md index f495ffbb..458675c0 100644 --- a/libs/astx-transpilers/README.md +++ b/libs/astx-transpilers/README.md @@ -135,14 +135,14 @@ example: ## Transpilers -ASTx includes transpilers that can convert your AST structures into Python code. -This makes it easy to generate executable Python from your abstract -representations. +ASTx includes a powerful transpiler system to convert your AST structures into +executable Python code. This is great for code generation, prototyping, or +building custom language tools. ```python from astx_transpilers.python_string import ASTxPythonTranspiler -# Using the same add_function from above +# Using the 'add_function' ASTx node from the example above transpiler = ASTxPythonTranspiler() python_code = transpiler.visit(add_function) print(python_code) @@ -155,9 +155,8 @@ def add(x: int, y: int) -> int: return (x + y) ``` -The transpiler handles various ASTx constructs including functions, variables, -expressions, and control flow. For comprehensive examples and detailed usage, -check out our [transpiler documentation](https://arxlang.github.io/astx). +For a deep dive into the architecture and more hands-on examples, check out our +**[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. --- From 215f29b3e2368dc53ae0098c3b4dce24b46c29ae Mon Sep 17 00:00:00 2001 From: Nagajaideep Date: Sat, 6 Sep 2025 12:14:04 +0530 Subject: [PATCH 3/8] linter test fix --- docs/index.md | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/docs/index.md b/docs/index.md index ff2ebe8a..809e0781 100644 --- a/docs/index.md +++ b/docs/index.md @@ -47,16 +47,17 @@ pip install astx ## 📖 Overview -ASTx is designed around two main concepts: +ASTx is designed around two primary concepts: -1. **Nodes**: each node represents a language construct (e.g., `Variable`, - `Function`, `IfStmt`) -2. **Tree**: nodes are organized hierarchically, forming an abstract - representation of the program +- **Nodes**: Each node represents a language construct (e.g., `Variable`, + `Function`, `IfStmt`). +- **Tree**: Nodes are organized hierarchically, forming an abstract + representation of the program structure. -ASTx also includes a simple transpiler for converting ASTx nodes to Python code -(text). This feature is for educational purposes, showing how a transpiler can -be implemented. +Additionally, ASTx provides a simple transpiler for converting ASTx nodes to +Python code (in text format). This feature is intended solely for educational +purposes, demonstrating how a transpiler from ASTx to any other language can be +implemented. --- @@ -67,7 +68,7 @@ be implemented. ```python import astx -# Define a simple function: add(x, y) -> x + y +# Define a simple function `add(x, y): return x + y` args = astx.Arguments( astx.Argument(name="x", type_=astx.Int32()), astx.Argument(name="y", type_=astx.Int32()), @@ -91,31 +92,32 @@ Use a transpiler to convert the AST to Python code: ```python from astx_transpilers.python_string import ASTxPythonTranspiler +# Transpile the AST to Python transpiler = ASTxPythonTranspiler() python_code = transpiler.visit(add_function) print(python_code) ``` -Output: +**Output:** ```python def add(x: int, y: int) -> int: return (x + y) ``` -### 3. Visualize ASTs +### 3. ASTx Visualization Features -ASTx supports multiple visualization formats: +ASTx offers multiple ways to visualize the AST structure: - YAML - JSON -- Graphical (PNG or ASCII) +- Graphical visualization (PNG or ASCII) -In a Jupyter Notebook, the default visualization is **PNG**. In a console, the -default is **ASCII**. +In a Jupyter Notebook, the default graphical visualization is PNG, while in a +console, the default is ASCII. -Example: +You can also print the AST structure in JSON or YAML format. For example: ```python print(add_function.to_json()) @@ -133,6 +135,7 @@ language tools. ```python from astx_transpilers.python_string import ASTxPythonTranspiler +# Using the 'add_function' ASTx node from the example above transpiler = ASTxPythonTranspiler() python_code = transpiler.visit(add_function) print(python_code) @@ -145,8 +148,8 @@ def add(x: int, y: int) -> int: return (x + y) ``` -For more examples, see the -**[transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. +For a deep dive into the architecture and more hands-on examples, check out our +**[full transpiler tutorial](tutorials/astx_transpiler_refactor_tutorial.md)**. --- @@ -157,7 +160,7 @@ Full documentation and examples: --- -## 🛠️ Contributing +## 🤝 Contributing Contributions are welcome! See the [Contributing Guide](https://astx.arxlang.org/contributing/) for details. From 8157ba7c20e75786d172e3f816a2dbfffc0e66df Mon Sep 17 00:00:00 2001 From: Nagajaideep Date: Sat, 6 Sep 2025 12:42:07 +0530 Subject: [PATCH 4/8] restored old index.md with new link --- docs/index.md | 207 +++++++++++++++----------------------------------- 1 file changed, 62 insertions(+), 145 deletions(-) diff --git a/docs/index.md b/docs/index.md index 809e0781..8eece95c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,173 +1,90 @@ -# ASTx: Abstract Syntax Tree Framework +# ASTx Library: A Versatile Toolkit for Language Representation -![CI](https://img.shields.io/github/actions/workflow/status/arxlang/astx/main.yaml?logo=github&label=CI) -[![Python Versions](https://img.shields.io/pypi/pyversions/astx)](https://pypi.org/project/astx/) -[![Package Version](https://img.shields.io/pypi/v/astx?color=blue)](https://pypi.org/project/astx/) -![License](https://img.shields.io/pypi/l/astx?color=blue) -![Discord](https://img.shields.io/discord/966124290464428042?logo=discord&color=blue) +ASTx is a groundbreaking library designed to encapsulate language components in +an agnostic and pythonic way. It provides a comprehensive set of classes and +functionalities, allowing developers to articulate the core elements of any +programming language. -ASTx is a Python library for representing and working with Abstract Syntax Trees -(ASTs). It provides a unified interface for building tools such as compilers, -interpreters, and transpilers. +- License: BSD 3 Clause +- Documentation: https://astx.arxlang.org -With ASTx, you can define language constructs, transform ASTs, generate code, -and build tools for analysis. +## Core Features -ASTx is not a lexer or parser. It can be used together with parsers to provide a -higher-level representation of the AST. +### 1. **Expressive Language Components** -It integrates with [IRx](https://github.com/arxlang/irx), enabling code -generation with **LLVM**. Currently, only a subset of ASTx nodes is supported, -but active development is underway. +ASTx offers a rich suite of classes to describe essential language constructs +such as: -**Note:** This project is under active development and not ready for production -use. +- `If` statements +- `For` loops +- `Function` declarations and usages +- Variables +- Data Types +- Operations +- And more ---- +These classes enable a concise and clear representation, providing an intuitive +way to model various programming constructs. -## 🚀 Features +### 2. **Symbol Table Class** -- Language-agnostic: works across different programming languages -- Extensible: add new nodes or extend existing ones -- Code generation: convert ASTs to Python code (other backends planned) -- Includes common constructs: variables, functions, control flow, types -- Initial Symbol Table implementation included +An integral part of ASTx, the Symbol Table class facilitates the translation of +ASTx expressions to other languages like LLVM-IR. This class acts as a mapping +layer, allowing a seamless connection between ASTx expressions and target +language representations. ---- +### 3. **Language Agnostic Design** -## 📦 Installation +Uniquely tailored to be independent of specific programming languages, ASTx +offers a flexible foundation. It strives to provide initial components that can +describe any programming language, giving users the freedom to work with +multiple languages effortlessly. -Install ASTx from PyPI: +### 4. **Integration with Projects like ARX-IR** -```bash -pip install astx -``` +ASTx has proven to be a vital tool in projects like ARX-IR, where it's leveraged +to translate Abstract Syntax Trees (AST) into LLVM-IR. This showcases the +library's adaptability and potential to serve as a foundational layer in various +applications. --- -## 📖 Overview - -ASTx is designed around two primary concepts: - -- **Nodes**: Each node represents a language construct (e.g., `Variable`, - `Function`, `IfStmt`). -- **Tree**: Nodes are organized hierarchically, forming an abstract - representation of the program structure. - -Additionally, ASTx provides a simple transpiler for converting ASTx nodes to -Python code (in text format). This feature is intended solely for educational -purposes, demonstrating how a transpiler from ASTx to any other language can be -implemented. - ---- - -## ✨ Usage - -### 1. Create an AST - -```python -import astx - -# Define a simple function `add(x, y): return x + y` -args = astx.Arguments( - astx.Argument(name="x", type_=astx.Int32()), - astx.Argument(name="y", type_=astx.Int32()), -) -fn_body = astx.Block() -fn_body.append( - astx.FunctionReturn( - value=astx.BinaryOp(op_code="+", lhs=astx.Variable("x"), rhs=astx.Variable("y")) - ) -) -add_function = astx.FunctionDef( - prototype=astx.FunctionPrototype(name="add", args=args, return_type=astx.Int32()), - body=fn_body, -) -``` - -### 2. Generate Code - -Use a transpiler to convert the AST to Python code: - -```python -from astx_transpilers.python_string import ASTxPythonTranspiler - -# Transpile the AST to Python -transpiler = ASTxPythonTranspiler() -python_code = transpiler.visit(add_function) - -print(python_code) -``` - -**Output:** +## New Transpiler System -```python -def add(x: int, y: int) -> int: - return (x + y) -``` +ASTx includes a powerful and redesigned transpiler system that allows you to +convert ASTx nodes into Python code. The new architecture separates the process +into two distinct steps: -### 3. ASTx Visualization Features +1. **ASTx → Python AST Objects**: Converts ASTx nodes into Python's built-in AST + objects. +2. **ASTx → Python Source Code**: Converts ASTx nodes directly into executable + Python code. -ASTx offers multiple ways to visualize the AST structure: +This separation improves maintainability, eliminates circular dependencies, and +makes the transpiler system more extensible. -- YAML -- JSON -- Graphical visualization (PNG or ASCII) - -In a Jupyter Notebook, the default graphical visualization is PNG, while in a -console, the default is ASCII. - -You can also print the AST structure in JSON or YAML format. For example: - -```python -print(add_function.to_json()) -print(add_function.to_yaml()) -``` - ---- - -## 🔄 Transpilers - -ASTx includes a transpiler system for converting AST structures into Python -code. This can be used for code generation, prototyping, or experimenting with -language tools. - -```python -from astx_transpilers.python_string import ASTxPythonTranspiler - -# Using the 'add_function' ASTx node from the example above -transpiler = ASTxPythonTranspiler() -python_code = transpiler.visit(add_function) -print(python_code) -``` - -**Output:** - -```python -def add(x: int, y: int) -> int: - return (x + y) -``` - -For a deep dive into the architecture and more hands-on examples, check out our -**[full transpiler tutorial](tutorials/astx_transpiler_refactor_tutorial.md)**. +For detailed usage instructions and examples, check out the +**[Transpiler Tutorial](tutorials/astx_transpiler_refactor_tutorial.md)**. --- -## 📚 Documentation +## Why Choose ASTx? -Full documentation and examples: -[https://arxlang.github.io/astx](https://arxlang.github.io/astx) +ASTx is not just a library; it's a robust framework that fosters creativity and +efficiency in language processing. Its pythonic design, combined with the power +to handle different language constructs, positions ASTx as an invaluable +resource for developers and researchers alike. ---- - -## 🤝 Contributing +Whether you're building a compiler, working on language translation, or +exploring new frontiers in programming language design, ASTx offers a reliable +and extensible toolkit to support your endeavors. -Contributions are welcome! See the -[Contributing Guide](https://astx.arxlang.org/contributing/) for details. - ---- +## Getting Started -## 📝 License +You can explore the ASTx library and dive into its capabilities by accessing the +official documentation. For those interested in contributing or seeking further +insights, the ASTx community provides extensive support and collaboration +opportunities. -ASTx is open-source software licensed under the **BSD-3-Clause License**. See -[LICENSE](LICENSE) for details. +Unlock the potential of language representation with ASTx, and join us in +shaping the future of programming languages. From 23fb1b508d8aa4358bb67ab55758122fc939fd72 Mon Sep 17 00:00:00 2001 From: Nagajaideep Date: Sun, 7 Sep 2025 15:41:19 +0530 Subject: [PATCH 5/8] gitignore: Add build artifacts and poetry build files to ignore list --- .gitignore | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.gitignore b/.gitignore index bdf0ff1e..db0ecca7 100644 --- a/.gitignore +++ b/.gitignore @@ -98,3 +98,15 @@ ENV/ # IDE settings .vscode/ .idea/ + +# Poetry/Build artifacts for library subdirectories +libs/astx/dist/ +libs/astx/build/ +libs/astx/*.egg-info/ +libs/astx-transpilers/dist/ +libs/astx-transpilers/build/ +libs/astx-transpilers/*.egg-info/ + +# README.md files copied during build process +libs/astx/README.md +libs/astx-transpilers/README.md From 5a1612ac555b4ff97e891566b4faedeb678cd036 Mon Sep 17 00:00:00 2001 From: Nagajaideep Date: Fri, 12 Sep 2025 19:48:40 +0530 Subject: [PATCH 6/8] chore: Sync all files after local build and doc updates --- .gitignore | 4 +--- .pre-commit-config.yaml | 1 + 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/.gitignore b/.gitignore index db0ecca7..ea3bfdd9 100644 --- a/.gitignore +++ b/.gitignore @@ -99,14 +99,12 @@ ENV/ .vscode/ .idea/ -# Poetry/Build artifacts for library subdirectories +# Ignore build artifacts and copied files libs/astx/dist/ libs/astx/build/ libs/astx/*.egg-info/ libs/astx-transpilers/dist/ libs/astx-transpilers/build/ libs/astx-transpilers/*.egg-info/ - -# README.md files copied during build process libs/astx/README.md libs/astx-transpilers/README.md diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index f916efba..2eac8c97 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -27,6 +27,7 @@ repos: entry: ./scripts/build.sh language: system pass_filenames: false + stages: [manual] - id: ruff-format name: ruff-format From 011a334f824fd253be8bbd9993590581ea88fe10 Mon Sep 17 00:00:00 2001 From: Nagajaideep Date: Fri, 12 Sep 2025 20:18:52 +0530 Subject: [PATCH 7/8] ci/cd fix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 458675c0..0d918170 100644 --- a/README.md +++ b/README.md @@ -155,7 +155,7 @@ def add(x: int, y: int) -> int: return (x + y) ``` -For a deep dive into the architecture and more hands-on examples, check out our +For a deep dive into the architecture and more hands-on examples, check out the **[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. --- From cc5029566ce89e4a688831872e5060f4a600604b Mon Sep 17 00:00:00 2001 From: Ivan Ogasawara Date: Fri, 12 Sep 2025 17:27:49 +0000 Subject: [PATCH 8/8] fix small issues --- .gitignore | 11 +++-------- .pre-commit-config.yaml | 1 - libs/astx-transpilers/README.md | 2 +- libs/astx/README.md | 25 +++++++++++++++++++++++++ 4 files changed, 29 insertions(+), 10 deletions(-) diff --git a/.gitignore b/.gitignore index ea3bfdd9..bfd8e3bf 100644 --- a/.gitignore +++ b/.gitignore @@ -100,11 +100,6 @@ ENV/ .idea/ # Ignore build artifacts and copied files -libs/astx/dist/ -libs/astx/build/ -libs/astx/*.egg-info/ -libs/astx-transpilers/dist/ -libs/astx-transpilers/build/ -libs/astx-transpilers/*.egg-info/ -libs/astx/README.md -libs/astx-transpilers/README.md +dist/ +build/ +*.egg-info/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 2eac8c97..f916efba 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -27,7 +27,6 @@ repos: entry: ./scripts/build.sh language: system pass_filenames: false - stages: [manual] - id: ruff-format name: ruff-format diff --git a/libs/astx-transpilers/README.md b/libs/astx-transpilers/README.md index 458675c0..0d918170 100644 --- a/libs/astx-transpilers/README.md +++ b/libs/astx-transpilers/README.md @@ -155,7 +155,7 @@ def add(x: int, y: int) -> int: return (x + y) ``` -For a deep dive into the architecture and more hands-on examples, check out our +For a deep dive into the architecture and more hands-on examples, check out the **[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. --- diff --git a/libs/astx/README.md b/libs/astx/README.md index 55c1d47e..0d918170 100644 --- a/libs/astx/README.md +++ b/libs/astx/README.md @@ -133,6 +133,31 @@ example: >>> print(add_function.to_yaml()) ``` +## Transpilers + +ASTx includes a powerful transpiler system to convert your AST structures into +executable Python code. This is great for code generation, prototyping, or +building custom language tools. + +```python +from astx_transpilers.python_string import ASTxPythonTranspiler + +# Using the 'add_function' ASTx node from the example above +transpiler = ASTxPythonTranspiler() +python_code = transpiler.visit(add_function) +print(python_code) +``` + +**Output:** + +```python +def add(x: int, y: int) -> int: + return (x + y) +``` + +For a deep dive into the architecture and more hands-on examples, check out the +**[full transpiler tutorial](docs/tutorials/astx_transpiler_refactor_tutorial.md)**. + --- ## 📚 Documentation