SQL Array methods to support multi choice fields #7239

labkey-matthewb · 2025-12-02T22:54:07Z

Rationale

SQL Array methods to support multi choice fields. Dialect methods for use by LabKey SQL as well as dataregion filters (NYI).

Related Pull Requests

Changes

checkpoint

SELECT 'a' as test, array_contains_all( ARRAY['A','X'], ARRAY['A','B'] ) as result UNION ALL SELECT 'b' as test, array_contains_any( ARRAY['A','X'], ARRAY['A','B'] ) as result UNION ALL SELECT 'c' as test, array_contains_none( ARRAY['A','X'], ARRAY['A','B'] ) as result UNION ALL SELECT 'd' as test, array_is_same( ARRAY['A','X'], ARRAY['A','B'] ) as result UNION ALL SELECT 'd' as test, array_is_same( ARRAY['A','B'], ARRAY['A','B'] ) as result

XingY · 2025-12-04T22:06:46Z

core/src/org/labkey/core/dialect/PostgreSql92Dialect.java

+        for (SQLFragment element : elements)
+        {
+            ret.append(separator);
+            ret.append(element);


Is there special encoding needed? For example a'b needs to be encoded into a''b?

Any encoding should be handled by whoever generated the elements. Those should be correct SQL already.

XingY · 2025-12-04T22:11:37Z

core/src/org/labkey/core/dialect/PostgreSql92Dialect.java

+            ret.append(element);
+            separator = ", ";
+        }
+        ret.append("]");


Do we always assume text[] type? I don't think this works for varchar[]. If the column is of type varchar[], the column needs to be casted to text[] first for the comparators to work:
SELECT * from vehicle.colors where tagsvarchar::text[] @> ARRAY['pastel', 'primary'];

Good question... We assume that any array we create would be text, and I think we an require that for module created tables as well. It would be tricky for us to handle varchar (say for attached schemas). Currently we don't track varchar and text separately they tend to just get merged as JdbcType.VARCHAR.

Worth adding a comment.

I wonder what happens when using VARCHAR columns e.g

tagsvarchar::text[] @> ARRAY[varcharColumn1, varcharColumn2'];

Maybe we need a TEXTARRAY() method? I'll think about it.

XingY · 2025-12-04T22:31:30Z

query/src/org/labkey/query/sql/Method.java

+    {
+        // (ELEMENT...)
+        postgresMethods.put("array_construct", new ArrayConstructMethod("array_construct"));
+


It would be useful to have a method that returns the number of elements in the array. Or a method that checks if array contains >1 elements. This would be useful for the story that support converting between single and multi value text choices.

textarray_construct()

XingY · 2025-12-08T22:05:56Z

core/src/org/labkey/core/dialect/PostgreSql92Dialect.java

+    @Override
+    public SQLFragment array_none_in_array(SQLFragment a, SQLFragment b)
+    {
+        return new SQLFragment(" NOT (").append(array_some_in_array(a, b)).append(")");


Should array_none_in_array also return empty values?

XingY · 2025-12-08T23:51:04Z

query/src/org/labkey/query/sql/Method.java

+
+        // not array_equals() because arrays are ordered, this is an unordered comparison
+        postgresMethods.put("array_is_same", new ArrayOperatorMethod("array_is_same", SqlDialect::array_same_array));
+        // Use "NOT array_is_same()" instead of something clumsy like "array_is_not_same()"


All of those methods only return records that are not empty. However, NOT array_op also only return non empty rows. I would think "Does Not include Any Of" should also return empty reocrds.

The spec calls for support for "Is Blank" and "Is not blank" methods. Maybe we should also have at least a array_is_empty method?

For VARCHAR we basically said we don't want both NULL and '', it's too confusing, so we always convert ''->NULL. I can see that NULL and [] are also super confusing. Perhaps we could always convert ''/blank/null->[] in this case?

I like the array_is_empty() idea.

Copilot

Pull request overview

This pull request introduces SQL array methods to support multi-choice fields in LabKey SQL and data region filters. The implementation adds dialect-specific methods for array operations, primarily targeting PostgreSQL.

Key Changes:

Added array literal syntax support (ARRAY[...] and TEXTARRAY[...]) to the SQL grammar
Implemented array operation methods (contains, containment checking, equality comparison) in SqlDialect with PostgreSQL-specific implementations
Added comprehensive unit tests for array operations covering various edge cases including NULL handling

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`query/src/org/labkey/query/sql/SqlBase.g`	Added ARRAY and TEXTARRAY tokens and grammar rules for array literal syntax
`query/src/org/labkey/query/sql/Method.java`	Introduced ArrayOperatorMethod, ArrayConstructMethod, and TextArrayConstructMethod classes; registered array methods for PostgreSQL; applied code formatting improvements
`query/src/org/labkey/query/QueryTestCase.jsp`	Added comprehensive unit tests for dialect array methods and SQL array operations
`core/src/org/labkey/core/dialect/PostgreSql92Dialect.java`	Implemented PostgreSQL-specific array operations using native operators (<@, &&, ANY, ALL)
`api/src/org/labkey/api/data/dialect/SqlDialect.java`	Added abstract array support methods to base SqlDialect class

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

query/src/org/labkey/query/QueryTestCase.jsp

Copilot · 2025-12-09T02:47:47Z

query/src/org/labkey/query/sql/Method.java

+        postgresMethods.put("array_contains_element", new ArrayOperatorMethod("array_contains_element",  (d,a,b) -> d.element_in_array(b,a)));
+        // Use "NOT array_contains_element()" instead of something clumsy like "array_does_not_contain()"
+
+        // (ARRAY, ARRAY)
+        postgresMethods.put("array_contains_all", new ArrayOperatorMethod("array_contains_all", (d,a,b) -> d.array_all_in_array(b,a)));


The lambda expressions on lines 1642, 1646 have inconsistent spacing. Line 1642 has (d,a,b) with no spaces after commas, while line 1647-1648 follow the file's convention with proper spacing. Add spaces after commas: (d, a, b)

Suggested change

postgresMethods.put("array_contains_element", new ArrayOperatorMethod("array_contains_element", (d,a,b) -> d.element_in_array(b,a)));

// Use "NOT array_contains_element()" instead of something clumsy like "array_does_not_contain()"

// (ARRAY, ARRAY)

postgresMethods.put("array_contains_all", new ArrayOperatorMethod("array_contains_all", (d,a,b) -> d.array_all_in_array(b,a)));

postgresMethods.put("array_contains_element", new ArrayOperatorMethod("array_contains_element", (d, a, b) -> d.element_in_array(b, a)));

// Use "NOT array_contains_element()" instead of something clumsy like "array_does_not_contain()"

// (ARRAY, ARRAY)

postgresMethods.put("array_contains_all", new ArrayOperatorMethod("array_contains_all", (d, a, b) -> d.array_all_in_array(b, a)));

Copilot · 2025-12-09T02:47:47Z

api/src/org/labkey/api/data/dialect/SqlDialect.java

+    // construct a sql array from SQLFragment elements
+    public SQLFragment array_construct(SQLFragment[] elements)
+    {
+        assert !supportsArrays();
+        throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement");
+    }
+
+    // element a is in array b
+    public SQLFragment element_in_array(SQLFragment a, SQLFragment b)
+    {
+        assert !supportsArrays();
+        throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement");
+    }
+
+    // element a is not in array b
+    public SQLFragment element_not_in_array(SQLFragment a, SQLFragment b)
+    {
+        assert !supportsArrays();
+        throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement");
+    }
+
+    // SET OPERATORS FOR ARRAY TYPE
+
+    // all elements of array a are contained in array b
+    public SQLFragment array_all_in_array(SQLFragment a, SQLFragment b)
+    {
+        assert !supportsArrays();
+        throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement");
+    }
+
+    // some elements of array a are contained in array b
+    public SQLFragment array_some_in_array(SQLFragment a, SQLFragment b)
+    {
+        assert !supportsArrays();
+        throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement");
+    }
+
+    // no elements of array a are contained in array b
+    public SQLFragment array_none_in_array(SQLFragment a, SQLFragment b)
+    {
+        assert !supportsArrays();
+        throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement");
+    }
+
+    // arrays a and b contain the same elements equivalent to (A all in B) AND (B all in A)
+    public SQLFragment array_same_array(SQLFragment a, SQLFragment b)
+    {
+        assert !supportsArrays();
+        throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement");
+    }
+
+    // array a and array b do not contain the same elements
+    public SQLFragment array_not_same_array(SQLFragment a, SQLFragment b)
+    {
+        assert !supportsArrays();
+        throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement");
+    }


The comments for these public API methods should use proper Javadoc format with /** instead of // for better documentation generation. Consider using proper Javadoc like:

/** * Construct a SQL array from SQLFragment elements. * @param elements The elements to include in the array * @return SQLFragment representing the array construction */

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

labkey-matthewb added 5 commits December 1, 2025 16:42

array methods

6b26a9b

checkpoint

Merge remote-tracking branch 'origin/develop' into fb_dialect_arrays

47f43aa

junit test

2a0007c

junit test

fd14ba1

labkey-matthewb requested a review from XingY December 3, 2025 20:23

XingY reviewed Dec 4, 2025

View reviewed changes

labkey-matthewb added 2 commits December 8, 2025 09:58

Merge remote-tracking branch 'origin/develop' into fb_dialect_arrays

bd43085

TEXTARRAY[]

6994b54

textarray_construct()

labkey-matthewb requested a review from XingY December 8, 2025 18:58

XingY reviewed Dec 8, 2025

View reviewed changes

XingY requested a review from Copilot December 9, 2025 02:37

Copilot started reviewing on behalf of XingY December 9, 2025 02:38 View session

Copilot AI reviewed Dec 9, 2025

View reviewed changes

labkey-matthewb and others added 2 commits December 9, 2025 20:49

Update query/src/org/labkey/query/QueryTestCase.jsp

239b3d8

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update query/src/org/labkey/query/QueryTestCase.jsp

98554f5

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

labkey-matthewb requested a review from XingY December 11, 2025 17:31

Merge remote-tracking branch 'origin/develop' into fb_dialect_arrays

4c94952

XingY approved these changes Dec 11, 2025

View reviewed changes

labkey-matthewb merged commit a84e761 into develop Dec 11, 2025
12 of 14 checks passed

labkey-matthewb deleted the fb_dialect_arrays branch December 11, 2025 19:48

SQL Array methods to support multi choice fields #7239

SQL Array methods to support multi choice fields #7239

Conversation

labkey-matthewb commented Dec 2, 2025

Rationale

Related Pull Requests

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants