Add create table support, venice deployer, and various fixes #182

jogrogan · 2025-12-12T17:37:40Z

This change implements the following:

Adds create table support
- Does lead to some awkwardness since you cannot construct a HoptimatorJdbcTable directly but the table must exist in the connection schema in order for the Deployer to create it. Worked around this using a temporary table structure, but I don't love this.
- It's also a little weird that Deployers will often need the same properties as are specified in the DB specific jdbc urls, we should make some way to passthrough this information rather than having to specify it in two places (in reference to the configmap update)
Adds a Venice Deployer & DeployerProvider that will get kicked off by CREATE TABLE VENICE."<my-store>".
- Includes create, update, and delete support
- Added several integration tests
Fixes Avro default values for null avro union ["null", "SqlType"]
- Without defaults, Venice schemas wouldn't be backwards compatible causing update to always fail
Fixes PipelineReconciler by adding missing failed state.
Made log lines consistent across DDL statements
One main issue with SPIs in Java is that when a dependency is pulled in, you also get the SPI. Meaning if we want an internal implementation of the Venice DeployerProvider, you can't override the OSS one. I added the ability for the DeploymentService to filter out top level SPIs in favor of a defined subclasses, should one exist.

Left various TODOs for future enhancements

ryannedolan

Really nice tests!

ryannedolan · 2025-12-13T17:42:42Z

hoptimator-jdbc/src/main/java/com/linkedin/hoptimator/jdbc/HoptimatorDdlExecutor.java

+
+    // TODO: Add support for populating new tables from a query as a one-time operation.
+    if (create.query != null) {
+      throw new DdlException(create, "Populating new tables is not currently supported.");


For us, I think CTAS would be the same as an MV, except without the job. Table triggers would work the same, and would backfill the table asynchronously. Maybe we can add REFRESH TABLE as well, which would make CTAS just a batch-only version of MV.

Yea it is similar but notably CTAS is meant as a one off backfill where as a MV is meant to have a running job to keep it in constant sync.
I guess it could be a trigger refreshed when executed but then not really again?

ryannedolan · 2025-12-13T17:53:13Z

hoptimator-jdbc/src/main/java/com/linkedin/hoptimator/jdbc/HoptimatorDdlExecutor.java

+          source = new Source(temporaryTable.databaseName(), tablePath, Collections.emptyMap());
+        }
+        deployers = DeploymentService.deployers(source, connection);
+        logger.info("Deleting table {}", tableName);


Wondering how we prevent a table from getting deleted if it's used in a view. Maybe we should add a Validator to that effect at some point.

Hmm that's interesting. Validating if an existing MV is sinking to the same table intended to be deleted is actually super easy. But if this table is used as the source of an MV, that's quite a bit harder today.

ryannedolan · 2025-12-13T17:53:43Z

hoptimator-jdbc/src/main/java/com/linkedin/hoptimator/jdbc/HoptimatorDdlExecutor.java

+  }
+
+  /**
+   * Temporary table implementation used during CREATE TABLE to provide row type information


I still don't love this tbh, this temporary table does become sticky and will likely cause problems if we are reusing connections. We could maybe have something in place that if a table resolves as a temporary table we force resolution of it?

ryannedolan · 2025-12-15T20:37:34Z

hoptimator-avro/src/main/java/com/linkedin/hoptimator/avro/AvroConverter.java

+            Object defaultValue = null;
+            // For unions containing null, defaults are specified in a specific way
+            if (innerField.isUnion() && innerField.isNullable()) {
+              defaultValue = Schema.Field.NULL_DEFAULT_VALUE;


ryannedolan · 2025-12-15T20:57:47Z

hoptimator-util/src/main/java/com/linkedin/hoptimator/util/DeploymentService.java


  public static <T extends Deployable> Collection<Deployer> deployers(T obj, Connection connection) {
-    return providers().stream()
+    Collection<DeployerProvider> providers = providers();


This is interesting. If downstream plugins are extending from upstream plugins, maybe we should split our dependencies into e.g. venice-core and venice-plugin (or whatever names make sense), s.t. we can depend on venice-core and not get the SPI service file?

Yea maybe, still some pros and cons. That'll get rid of some of the potential weirdness caused by only choosing subclasses (if there are any?) but still won't allow inheritance and overrides. Say we have this VeniceDeployer, we want to expose it internally but drop in a different ControllerClient implementation, we still wouldn't be able to extend this Deployer without pulling in the SPI (with that being said we could have a 3rd BaseClass of sorts existing in a different module). Idk both don't feel great to me.

ryannedolan · 2025-12-15T21:01:55Z

hoptimator-venice/src/main/java/com/linkedin/hoptimator/venice/VeniceDeployer.java

+
+  @Override
+  public void restore() {
+    // TODO: Restoration can be complicated


Maybe we can use connection.warn(msg) here.

jogrogan force-pushed the jogrogan/createTable branch 5 times, most recently from 71e7c4f to 8311ecd Compare December 12, 2025 20:47

jogrogan requested a review from ryannedolan December 15, 2025 15:00

ryannedolan approved these changes Dec 15, 2025

View reviewed changes

jogrogan added 7 commits January 6, 2026 15:15

Add base CREATE TABLE

c10b658

Add Venice Deployer

332177b

Fix PipelineReconciler and AvroConverter defaults

e1d0b31

Add drop tests

4056ce6

Fix failing tests caused by specify closing connections prematurely

07ef35b

Add back some comments about copy-pasted code

0465e26

Add log

0971ac1

jogrogan force-pushed the jogrogan/createTable branch from d05859a to 0971ac1 Compare January 6, 2026 20:16

jogrogan merged commit 1ea853f into main Jan 7, 2026
1 check passed

jogrogan deleted the jogrogan/createTable branch January 7, 2026 16:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add create table support, venice deployer, and various fixes #182

Add create table support, venice deployer, and various fixes #182

Uh oh!

jogrogan commented Dec 12, 2025 •

edited

Loading

Uh oh!

ryannedolan left a comment

Uh oh!

ryannedolan Dec 13, 2025

Uh oh!

jogrogan Dec 18, 2025 •

edited

Loading

Uh oh!

ryannedolan Dec 13, 2025

Uh oh!

jogrogan Dec 18, 2025

Uh oh!

ryannedolan Dec 13, 2025

Uh oh!

jogrogan Dec 18, 2025

Uh oh!

ryannedolan Dec 15, 2025

Uh oh!

ryannedolan Dec 15, 2025

Uh oh!

jogrogan Dec 18, 2025 •

edited

Loading

Uh oh!

ryannedolan Dec 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add create table support, venice deployer, and various fixes #182

Add create table support, venice deployer, and various fixes #182

Uh oh!

Conversation

jogrogan commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryannedolan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jogrogan Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jogrogan Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jogrogan commented Dec 12, 2025 •

edited

Loading

jogrogan Dec 18, 2025 •

edited

Loading

jogrogan Dec 18, 2025 •

edited

Loading