Data fusion changes

[sandeshkr419]

Description

Version forced to 3.3.0.

Testing:

./gradlew :integ-test:integTest --tests "org.opensearch.sql.calcite.clickbench.CalcitePPLClickBenchIT" -Dtests.method="testDataFusion" -Dtests.cluster=localhost:9200 -Dtests.rest.cluster=localhost:9200 -DignorePrometheus=true -Dtests.clustername=opensearch -Dtests.output=true

    Total
    42
     queries succeed. Average duration:
    139
     ms


    Total:
    42
     | Passed:
    25
     | Failed (200):
    8
     | Failed (non-200):
    9

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[gemini-code-assist]

Summary of Changes

Hello @sandeshkr419, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the PPL query engine's capabilities by introducing a new Unified Query API and several powerful commands and functions. The changes focus on improving data fusion, query planning, and optimization, while also addressing type handling, error reporting, and resource management. These updates aim to provide a more robust and versatile analytical experience for users.

Highlights

• Unified Query API: Introduced a new api module with a UnifiedQueryPlanner class, providing a high-level, declarative API for parsing and analyzing PPL queries into Calcite RelNode logical plans. This simplifies integration for external systems like Apache Spark.
• New PPL Commands: Added support for several new PPL commands: multisearch to combine results from multiple subsearches, replace for text replacement with wildcard and regex support, and streamstats for calculating cumulative or rolling statistics over event streams.
• New PPL Functions: Implemented new collection and JSON functions: mvappend to append elements to an array, map_append and map_remove for map manipulation, and json_extract_all to extract all fields from JSON into a map.
• Improved Query Planning and Optimization: Enhanced the Calcite query planner with a new PPLAggGroupMergeRule for optimizing aggregate group fields. Significant improvements to fillnull, bin, rex, and timechart commands, including better type compatibility checks, named capture group support for rex, and dynamic rate calculation for timechart's per_second functions.
• Subquery and Join Limit Settings: Introduced new settings (plugins.ppl.subsearch.maxout and plugins.ppl.join.subsearch_maxout) to control the maximum number of rows returned by subsearches and subsearches involved in join operations, preventing excessive resource consumption.
• Enhanced Type Coercion and Error Handling: Refactored type coercion logic for improved accuracy and added stricter validation for fillnull type compatibility and rex named capture group syntax, providing clearer error messages.
• Build and Dependency Updates: Updated Apache Calcite to version 1.41.0, integrated Substrait dependencies, and consolidated Maven snapshot repository URLs. Also included dependency version bumps for Jackson and Protobuf, and added a dependency substitution for commons-lang to address a CVE.

Ignored Files

• Ignored by pattern: .github/workflows/** (8)
  • .github/workflows/datafusion-e2e-test.yml
  • .github/workflows/delete_backport_branch.yml
  • .github/workflows/maven-publish-modules.yml
  • .github/workflows/maven-publish.yml
  • .github/workflows/publish-async-query-core.yml
  • .github/workflows/publish-grammar-files.yml
  • .github/workflows/sql-test-and-build-workflow.yml
  • .github/workflows/stalled.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant set of changes, primarily focused on a new "Data Fusion" or "Unified Query" API, along with a major Calcite version upgrade and the introduction of Substrait dependencies. It also adds several new PPL commands (multisearch, replace, streamstats) and functions, and includes numerous refactorings and documentation updates.

My review has identified a few critical and high-severity issues that should be addressed before merging:

• A hardcoded change in QueryService.java enables the Calcite engine for all query types, including SQL, which is not yet fully supported and could cause breakages.
• The mapping of IP and binary data types to BIGINT in OpenSearchTypeFactory.java is a breaking change that will affect functions relying on these types.
• There is a minor duplication in the root build.gradle file that should be cleaned up for maintainability.

Overall, the introduction of the Unified Query API and the refactoring efforts (like QualifiedNameResolver and SchemaUnifier) are positive steps towards a more robust and extensible query engine. The documentation updates are also very thorough and helpful. Once the identified issues are resolved, this will be a strong contribution.

[gemini-code-assist]

The shouldUseCalcite method is hardcoded to return true. Based on the commented-out code and the comment // Calcite is not available for SQL query now., it seems this change is intended for development or testing purposes. Merging this will enable the Calcite engine for all query types, including SQL, which could lead to unexpected behavior or breakages for SQL queries. This should be reverted to the original logic before merging.

Suggested change

	private boolean shouldUseCalcite(QueryType queryType) {
	return isCalciteEnabled(settings) && queryType == QueryType.PPL;
	return true;//isCalciteEnabled(settings) && queryType == QueryType.PPL;
	}
	private boolean shouldUseCalcite(QueryType queryType) {
	return isCalciteEnabled(settings) && queryType == QueryType.PPL;
	}

[gemini-code-assist]

The IP data type is now being mapped to BIGINT instead of the user-defined EXPR_IP type. The comment suggests this is for Substrait compatibility. However, this is a breaking change that will likely cause functions that operate on IP addresses, such as CIDRMATCH, to fail because they will receive a BIGINT instead of the expected IP type. This change should be made conditional or handled in a way that doesn't break existing functionality for non-Substrait use cases.

Suggested change

	return TYPE_FACTORY.createSqlType(SqlTypeName.BIGINT, nullable);
	// return TYPE_FACTORY.createUDT(ExprUDT.EXPR_IP, nullable);
	return TYPE_FACTORY.createUDT(ExprUDT.EXPR_IP, nullable);

[gemini-code-assist]

The legacy binary data type is now being mapped to BIGINT instead of the user-defined EXPR_BINARY type. Similar to the change for the IP type, this is a breaking change motivated by Substrait compatibility. This could cause issues for any functionality that relies on the binary type. This change should be conditional to avoid breaking existing behavior.

Suggested change

	// return TYPE_FACTORY.createUDT(ExprUDT.EXPR_BINARY, nullable);
	} else if (fieldType.legacyTypeName().equalsIgnoreCase("timestamp")) {
	return TYPE_FACTORY.createUDT(ExprUDT.EXPR_BINARY, nullable);

[gemini-code-assist]

The legacy ip data type is also being mapped to BIGINT. This is a breaking change for the same reasons as the IP enum case. This should be reverted to use the EXPR_IP UDT to maintain compatibility.

Suggested change

	return TYPE_FACTORY.createSqlType(SqlTypeName.BIGINT, nullable);
	// return TYPE_FACTORY.createUDT(ExprUDT.EXPR_IP, nullable);
	return TYPE_FACTORY.createUDT(ExprUDT.EXPR_IP, nullable);

[gemini-code-assist]

These force declarations for avatica-core and slf4j-api are duplicates of lines 158-159. The redundant lines can be removed for better maintainability.