Skip to content

upgrade to iceberg 1.10.1 #784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
kevinjqliu wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

upgrade to iceberg 1.10.1 #784

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
9fe9467
upgrade to iceberg 1.10.1
kevinjqliu Jan 15, 2026
41142e2
api changes
kevinjqliu Jan 15, 2026
cb4405d
use best available iceberg-hive-runtime
kevinjqliu Jan 15, 2026
04fa576
upgrade avro version
kevinjqliu Jan 15, 2026
c04f9ce
to run with java 17
kevinjqliu Jan 15, 2026
a2d6484
tests
kevinjqliu Jan 15, 2026
34f3a07
fix avro
kevinjqliu Jan 15, 2026
89247c9
dont change notebook
kevinjqliu Jan 15, 2026
d3f5674
fix tests
kevinjqliu Jan 15, 2026
a4b2a8c
use iceberg 1.9.2
kevinjqliu Jan 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 75 additions & 4 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@
<project.build.outputTimestamp>2025-01-01T00:00:00Z</project.build.outputTimestamp>
<maven.compiler.target>8</maven.compiler.target>
<maven-compiler-plugin.version>3.13.0</maven-compiler-plugin.version>
<avro.version>1.11.4</avro.version>
<avro.version>1.12.0</avro.version>
<log4j.version>2.22.0</log4j.version>
<junit.version>5.11.4</junit.version>
<junit.platform.runner.version>1.11.4</junit.platform.runner.version>
Expand All @@ -87,7 +87,8 @@
<scala.binary.version>2.12</scala.binary.version>
<spark.version>3.4.2</spark.version>
<spark.version.prefix>3.4</spark.version.prefix>
<iceberg.version>1.4.2</iceberg.version>
<iceberg.version>1.9.2</iceberg.version>
<iceberg.hive.runtime.version>1.7.2</iceberg.hive.runtime.version>
<delta.version>2.4.0</delta.version>
<paimon.version>1.2.0</paimon.version>
<jackson.version>2.18.2</jackson.version>
Expand Down Expand Up @@ -243,6 +244,11 @@
<groupId>org.openjdk.jol</groupId>
<artifactId>jol-core</artifactId>
</exclusion>
<!-- Exclude Parquet to enforce version 1.15+ required by Iceberg 1.10.1 -->
<exclusion>
<groupId>org.apache.parquet</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
Expand All @@ -262,19 +268,38 @@
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
<!-- Exclude Parquet to enforce version 1.15+ required by Iceberg 1.10.1 -->
<exclusion>
<groupId>org.apache.parquet</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-sync-common</artifactId>
<version>${hudi.version}</version>
<scope>provided</scope>
<exclusions>
<!-- Exclude Parquet to enforce version 1.15+ required by Iceberg 1.10.1 -->
<exclusion>
<groupId>org.apache.parquet</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-spark${spark.version.prefix}-bundle_${scala.binary.version}</artifactId>
<version>${hudi.version}</version>
<scope>test</scope>
<exclusions>
<!-- Exclude Parquet to enforce version 1.15+ required by Iceberg 1.10.1 -->
<exclusion>
<groupId>org.apache.parquet</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hudi</groupId>
Expand All @@ -283,11 +308,25 @@
<classifier>tests</classifier>
<type>test-jar</type>
<scope>test</scope>
<exclusions>
<!-- Exclude Parquet to enforce version 1.15+ required by Iceberg 1.10.1 -->
<exclusion>
<groupId>org.apache.parquet</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-java-client</artifactId>
<version>${hudi.version}</version>
<exclusions>
<!-- Exclude Parquet to enforce version 1.15+ required by Iceberg 1.10.1 -->
<exclusion>
<groupId>org.apache.parquet</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>

<!-- Iceberg -->
Expand Down Expand Up @@ -315,7 +354,7 @@
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-hive-runtime</artifactId>
<version>${iceberg.version}</version>
<version>${iceberg.hive.runtime.version}</version>
</dependency>

<!-- Delta -->
Expand Down Expand Up @@ -748,6 +787,22 @@
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<trimStackTrace>false</trimStackTrace>
<forkedProcessExitTimeoutInSeconds>120</forkedProcessExitTimeoutInSeconds>
<argLine>
--add-opens=java.base/java.lang=ALL-UNNAMED
Copy link
Contributor

@the-other-tim-brown the-other-tim-brown Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are all of these args required?

--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
<plugin>
Expand All @@ -767,7 +822,23 @@
<reuseForks>false</reuseForks>
<forkCount>6</forkCount>
<trimStackTrace>false</trimStackTrace>
<argLine>-Xmx1500m</argLine>
<argLine>
-Xmx1500m
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
</argLine>
<forkedProcessExitTimeoutInSeconds>120</forkedProcessExitTimeoutInSeconds>
</configuration>
</plugin>
Expand Down
4 changes: 2 additions & 2 deletions xtable-aws/src/main/resources/META-INF/LICENSE-bundled
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,8 +226,8 @@ org.apache.httpcomponents:fluent-hc:4.4.1
org.apache.hudi:hudi-common:0.14.0
org.apache.hudi:hudi-java-client:0.14.0
org.apache.hudi:hudi-timeline-service:0.14.0
org.apache.iceberg:iceberg-api:1.4.2
org.apache.iceberg:iceberg-core:1.4.2
org.apache.iceberg:iceberg-api:1.10.1
org.apache.iceberg:iceberg-core:1.10.1
org.apache.logging.log4j:log4j-1.2-api:2.22.0
org.apache.logging.log4j:log4j-api:2.22.0
org.javassist:javassist:3.25.0-GA
Expand Down
20 changes: 12 additions & 8 deletions xtable-core/src/main/java/org/apache/xtable/hudi/HudiFileStatsExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,14 +203,18 @@ private static ColumnStat getColumnStatFromHudiStat(
if (field.getSchema().getDataType() == InternalType.DECIMAL) {
int scale =
(int) field.getSchema().getMetadata().get(InternalSchema.MetadataKey.DECIMAL_SCALE);
minValue =
minValue instanceof ByteBuffer
? convertBytesToBigDecimal((ByteBuffer) minValue, scale)
: ((BigDecimal) minValue).setScale(scale, RoundingMode.UNNECESSARY);
maxValue =
maxValue instanceof ByteBuffer
? convertBytesToBigDecimal((ByteBuffer) maxValue, scale)
: ((BigDecimal) maxValue).setScale(scale, RoundingMode.UNNECESSARY);
if (minValue != null) {
minValue =
minValue instanceof ByteBuffer
? convertBytesToBigDecimal((ByteBuffer) minValue, scale)
: ((BigDecimal) minValue).setScale(scale, RoundingMode.UNNECESSARY);
}
if (maxValue != null) {
maxValue =
maxValue instanceof ByteBuffer
? convertBytesToBigDecimal((ByteBuffer) maxValue, scale)
: ((BigDecimal) maxValue).setScale(scale, RoundingMode.UNNECESSARY);
}
}
return getColumnStatFromValues(
minValue,
Expand Down
5 changes: 2 additions & 3 deletions xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelSchemaExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,16 +66,15 @@ public InternalSchema toInternalSchema(StructType structType) {
return toInternalSchema(structType, null, false, null, null);
}

String trimmedTypeName = "";
InternalType type = null;

private InternalSchema toInternalSchema(
DataType dataType,
String parentPath,
boolean nullable,
String comment,
FieldMetadata originalMetadata) {

String trimmedTypeName = "";
InternalType type = null;
Map<InternalSchema.MetadataKey, Object> metadata = null;
List<InternalField> fields = null;

Expand Down
2 changes: 1 addition & 1 deletion xtable-core/src/test/java/org/apache/xtable/TestIcebergTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@ private DataFile writeAndGetDataFile(List<Record> records, StructLike partitionK
DataWriter<Record> dataWriter =
Parquet.writeData(file)
.schema(icebergTable.schema())
.createWriterFunc(GenericParquetWriter::buildWriter)
.createWriterFunc(GenericParquetWriter::create)
.overwrite()
.withSpec(icebergTable.spec())
.withPartition(partitionKey)
Expand Down
2 changes: 1 addition & 1 deletion xtable-core/src/test/java/org/apache/xtable/iceberg/TestIcebergConversionSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,7 +432,7 @@ private DataFile generateTestDataFile(int partition, Table table, String filePat
DataWriter<GenericRecord> dataWriter =
Parquet.writeData(table.io().newOutputFile(filePath))
.schema(csSchema)
.createWriterFunc(GenericParquetWriter::buildWriter)
.createWriterFunc(GenericParquetWriter::create)
.overwrite()
.withSpec(table.spec())
.withPartition(partitionInfo)
Expand Down
6 changes: 1 addition & 5 deletions xtable-core/src/test/java/org/apache/xtable/iceberg/TestIcebergDataHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -333,11 +333,7 @@ private Object generateRandomValueForType(
case STRUCT:
return generateInsertRecord(timeLowerBound, timeUpperBound, fieldType.asStructType());
case UUID:
UUID uuid = UUID.randomUUID();
ByteBuffer byteBuffer = ByteBuffer.wrap(new byte[16]);
byteBuffer.putLong(uuid.getMostSignificantBits());
byteBuffer.putLong(uuid.getLeastSignificantBits());
return byteBuffer.array();
return UUID.randomUUID();
case LIST:
Types.ListType listType = (Types.ListType) fieldType;
int listSize = RANDOM.nextInt(5) + 1;
Expand Down
5 changes: 3 additions & 2 deletions xtable-core/src/test/java/org/apache/xtable/iceberg/TestIcebergSchemaSync.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ public void testUpdateListElement() {

verify(mockUpdateSchema).deleteColumn("array_field.element.element_string");
verify(mockUpdateSchema)
.addColumn("array_field.element", "element_double", Types.DoubleType.get(), null);
.addColumn("array_field.element", "element_double", Types.DoubleType.get(), (String) null);
verify(mockUpdateSchema).commit();
}

Expand All @@ -194,7 +194,8 @@ public void testUpdateMapValue() {

verify(mockUpdateSchema).deleteColumn("map_field.value.value_string");
verify(mockUpdateSchema)
.addRequiredColumn("map_field.value", "value_double", Types.DoubleType.get(), null);
.addRequiredColumn(
"map_field.value", "value_double", Types.DoubleType.get(), (String) null);
verify(mockUpdateSchema).commit();
}

Expand Down
25 changes: 14 additions & 11 deletions xtable-core/src/test/resources/schemas/basic_schema.avsc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,20 +72,23 @@
},
{
"name": "bytes_field",
"type": "bytes",
"default": ""
"type": ["null", "bytes"],
"default": null
},
{
"name": "decimal_field",
"type": {
"type": "fixed",
"name": "decimal_field_testing",
"size": 10,
"logicalType": "decimal",
"precision": 20,
"scale": 2
},
"default": "\u0000"
"type": [
"null",
{
"type": "fixed",
"name": "decimal_field_testing",
"size": 10,
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
],
"default": null
},
{
"name": "nested_record",
Expand Down
4 changes: 2 additions & 2 deletions xtable-hive-metastore/src/main/resources/META-INF/LICENSE-bundled
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,8 +225,8 @@ org.apache.httpcomponents:fluent-hc:4.4.1
org.apache.hudi:hudi-common:0.14.0
org.apache.hudi:hudi-java-client:0.14.0
org.apache.hudi:hudi-timeline-service:0.14.0
org.apache.iceberg:iceberg-api:1.4.2
org.apache.iceberg:iceberg-core:1.4.2
org.apache.iceberg:iceberg-api:1.10.1
org.apache.iceberg:iceberg-core:1.10.1
org.apache.logging.log4j:log4j-1.2-api:2.22.0
org.apache.logging.log4j:log4j-api:2.22.0
io.delta:delta-core_2.12:2.4.0
Expand Down
6 changes: 3 additions & 3 deletions ...e-hudi-support/xtable-hudi-support-extensions/src/main/resources/META-INF/LICENSE-bundled
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,9 +233,9 @@ org.apache.hudi:hudi-client-common:0.14.0
org.apache.hudi:hudi-common:0.14.0
org.apache.hudi:hudi-timeline-service:0.14.0
org.apache.hudi:hudi-sync-common:0.14.0
org.apache.iceberg:iceberg-api:1.4.2
org.apache.iceberg:iceberg-core:1.4.2
org.apache.iceberg:iceberg-parquet:1.4.2
org.apache.iceberg:iceberg-api:1.10.1
org.apache.iceberg:iceberg-core:1.10.1
org.apache.iceberg:iceberg-parquet:1.10.1
org.apache.logging.log4j:log4j-api:2.22.0
org.apache.logging.log4j:log4j-1.2-api:2.22.0
org.roaringbitmap:RoaringBitmap:0.9.47
Expand Down
Loading