upgrade pg_walstream version to 0.5.1

[isdaniel]

• Updated all test cases in replica_identity_tests.rs, sqlite_comprehensive_tests.rs, sqlite_destination_tests.rs, and where_clause_fix_tests.rs to replace serde_json::Value with pg_walstream::ColumnValue for consistency and improved type safety.
• Modified the format_value function to format ColumnValue types correctly in SQL statements.
• Introduced a new ReplicationActor to manage the LogicalReplicationStream in a dedicated thread, ensuring thread safety and proper command handling.
• Enhanced the actor's command structure to support starting the replication stream, reading events, sending feedback, and gracefully stopping the actor.
• Refactor transaction manager and tests: extract common utilities for SQL formatting and transaction handling

[gemini-code-assist]

Summary of Changes

Hello @isdaniel, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on a significant upgrade of the pg_walstream library, which necessitated a widespread refactoring to adapt to its new ColumnValue type. The changes streamline how column data is represented and formatted into SQL statements, enhancing type safety and robustness. The update also includes a new actor for managing the replication stream, improving the overall architecture and reliability of data processing and SQL generation across different database destinations.

Highlights

• Dependency Upgrade: Upgraded the pg_walstream dependency to version 0.5.1, bringing in new functionalities and API changes, and updated the chrono dependency to 0.4.44.
• Type System Modernization: Replaced serde_json::Value with pg_walstream::ColumnValue across the codebase, particularly in RowData and ChangeEvent structures, to enhance type safety and consistency when handling PostgreSQL column data.
• Robust SQL Formatting: Refactored the format_value function in TransactionManager to correctly handle ColumnValue types, implementing destination-specific SQL literal formatting, including hex encoding for binary data, heuristic type detection for text (numbers, booleans), and robust escaping to prevent SQL injection.
• Improved Test Coverage: Introduced a new common test utility module and added comprehensive unit tests for the format_value function, covering various data types, escaping rules, and SQL injection prevention scenarios, significantly improving the reliability of SQL generation.
• Logical Replication Stream Management: Implemented a new ReplicationActor to manage the LogicalReplicationStream in a dedicated thread, enhancing thread safety and providing a structured approach for handling stream commands like starting, reading events, sending feedback, and stopping.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• Cargo.lock
  • Updated pg_walstream to 0.5.1 and chrono to 0.4.44.
  • Removed serde_json from pg_walstream's transitive dependencies.
• Cargo.toml
  • Updated pg_walstream dependency to 0.5.1.
• pg2any-lib/src/client.rs
  • Removed direct calls to stream.send_feedback() and stream.current_lsn(), as these functionalities are now managed by the new ReplicationActor.
• pg2any-lib/src/lib.rs
  • Removed the export of format_postgres_timestamp, reflecting changes in the pg_walstream API.
• pg2any-lib/src/transaction_manager.rs
  • Imported ColumnValue for type-safe data handling.
  • Refactored generate_insert_sql, generate_update_sql, and build_where_clause to directly use RowData::iter(), eliminating intermediate HashMap conversions.
  • Rewrote format_value to accept ColumnValue, implementing detailed SQL literal formatting for Null, Text (with heuristic type detection and escaping), and Binary (with hex encoding).
  • Added extensive unit tests for format_value to ensure correct SQL literal generation and prevent injection vulnerabilities.
• pg2any-lib/tests/common/mod.rs
  • Added a new module containing shared test utilities, including wrap_in_transaction and format_column_value for consistent test data handling and SQL formatting in tests.
• pg2any-lib/tests/destination_integration_tests.rs
  • Updated test data creation to use ColumnValue instead of serde_json::Value.
• pg2any-lib/tests/event_type_refactor_tests.rs
  • Updated test data creation to use ColumnValue instead of serde_json::Value.
• pg2any-lib/tests/metrics_logical_tests.rs
  • Updated test data creation to use ColumnValue instead of serde_json::Value.
• pg2any-lib/tests/mysql_edge_cases_tests.rs
  • Updated test data creation to use ColumnValue instead of serde_json::Value and adjusted assertions.
• pg2any-lib/tests/mysql_error_handling_simple_tests.rs
  • Updated test data creation to use ColumnValue instead of serde_json::Value.
• pg2any-lib/tests/mysql_where_clause_fix_tests.rs
  • Updated test data creation to use ColumnValue instead of serde_json::Value and adjusted assertions to reflect ColumnValue usage.
• pg2any-lib/tests/replica_identity_tests.rs
  • Updated test data creation to use ColumnValue instead of serde_json::Value.
• pg2any-lib/tests/sqlite_comprehensive_tests.rs
  • Updated test data creation to use ColumnValue and refactored event_to_sql and format_value helpers to align with the new ColumnValue type and utilize shared utilities.
• pg2any-lib/tests/sqlite_destination_tests.rs
  • Updated test data creation to use ColumnValue and refactored event_to_sql and format_value helpers to align with the new ColumnValue type and utilize shared utilities.
• pg2any-lib/tests/where_clause_fix_tests.rs
  • Updated test data creation to use ColumnValue instead of serde_json::Value and adjusted assertions to reflect ColumnValue usage.

Activity

• The author initiated a dependency upgrade for pg_walstream to version 0.5.1.
• A significant refactoring effort was undertaken to replace serde_json::Value with the more specific pg_walstream::ColumnValue across data structures and SQL formatting logic.
• New test utilities were introduced in pg2any-lib/tests/common/mod.rs to streamline testing with the new ColumnValue type.
• Extensive unit tests were added and updated in pg2any-lib/src/transaction_manager.rs and various integration test files to ensure the correctness and security of SQL literal formatting.
• The PR description indicates the introduction of a ReplicationActor for managing the logical replication stream, suggesting a broader architectural improvement related to the pg_walstream upgrade.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request upgrades pg_walstream to version 0.5.1 and refactors the codebase to use pg_walstream::ColumnValue instead of serde_json::Value, which improves type handling and SQL generation efficiency. However, the refactored SQL generation introduces or maintains several high-severity SQL injection vulnerabilities due to improper escaping of database identifiers (schema, table, and column names) sourced from the PostgreSQL replication stream. Furthermore, high-severity data correctness issues were found in the new format_value logic concerning how numeric and boolean values are heuristically detected from text, potentially leading to data corruption. Inconsistencies in a new test helper were also noted.

[isdaniel]

/gemini review

[gemini-code-assist]

Code Review

This pull request upgrades pg_walstream to version 0.5.1 and refactors the codebase to use pg_walstream::ColumnValue instead of serde_json::Value. This is a great improvement for type safety and consistency. The SQL generation logic in TransactionManager has been significantly improved, making it more generic, robust, and easier to maintain by removing duplicated code. The addition of a comprehensive test suite for SQL value formatting and identifier quoting is also a major plus. The changes are well-executed and improve the overall quality of the code. I've found one minor issue in a test helper function that I've commented on.