Migrate surfaces over to binary format by jonathan-eq · Pull Request #12840 · equinor/ert

jonathan-eq · 2026-02-11T08:23:18Z

Issue
Resolves #12824

Approach
Before this commit, surfaces are read by ert in ascii format. This commit changes the default to be binary, with the option to override by setting SURFACE .... FORMAT:ASCII in config. This will retain the SURFACE old behavior.

(Screenshot of new behavior in GUI if applicable)

PR title captures the intent of the changes, and is fitting for release notes.
Added appropriate release note label
Commit history is consistent and clean, in line with the contribution guidelines.
Make sure unit tests pass locally after every commit (git rebase -i main --exec 'just rapid-tests')

When applicable

When there are user facing changes: Updated documentation
New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
Large PR: Prepare changes in small commits for more convenient review
Bug fix: Add regression test for the bug
Bug fix: Add backport label to latest release (format: 'backport release-branch-name')

src/ert/config/surface_config.py

codecov-commenter · 2026-02-11T09:23:10Z

Codecov Report

❌ Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.11%. Comparing base (a64ca23) to head (1a079ec).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/ert/config/surface_config.py	91.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12840      +/-   ##
==========================================
- Coverage   90.12%   90.11%   -0.02%     
==========================================
  Files         447      447              
  Lines       30946    30967      +21     
==========================================
+ Hits        27890    27905      +15     
- Misses       3056     3062       +6

Flag	Coverage Δ
cli-tests	`36.79% <62.50%> (+0.01%)`	⬆️
gui-tests	`67.23% <37.50%> (-0.03%)`	⬇️
performance-and-unit-tests	`76.79% <91.66%> (-0.01%)`	⬇️
test	`45.91% <20.83%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jonathan-eq · 2026-02-11T11:13:55Z

src/ert/config/surface_config.py

+        file_format = SurfaceFileFormat.BINARY
        try:
-            surf = IrapSurface.from_ascii_file(Path(base_surface))
+            try:
+                surf = IrapSurface.from_binary_file(Path(base_surface))
+                logger.info("Loaded surface in binary format")
+            except Exception:
+                surf = IrapSurface.from_ascii_file(Path(base_surface))
+                ConfigWarning.warn(ASCII_SURFACE_WARNING_MESSAGE)
+                file_format = SurfaceFileFormat.ASCII


The idea here is that if you have binary on the initial validation, then you stick with the binary version when writing/reading from runpath. If you still use ascii, you get the warning, but ert will continue as before, thus not breaking any forward models that still read/write ascii surfaces

src/ert/config/surface_config.py

codspeed-hq · 2026-02-11T12:54:11Z

Merging this PR will not alter performance

✅ 35 untouched benchmarks

_{Comparing jonathan-eq:surface-warning (49e32d4) with main (f684095)}

frode-aarstad · 2026-02-12T06:58:18Z

src/ert/config/surface_config.py

        super().__init__(msg)


+ASCII_SURFACE_WARNING_MESSAGE = (


We should probably add something about this in the docs aswell

frode-aarstad · 2026-02-12T07:07:05Z

tests/ert/unit_tests/config/test_surface_config.py

    )
+
+
+@pytest.mark.parametrize("is_ascii_surface", [True, False])


Could we add a test that tests if we read ascii we also write ascii

Yes, added 👌

xjules · 2026-02-13T09:38:43Z

src/ert/config/surface_config.py



+ASCII_SURFACE_WARNING_MESSAGE = (
+    "From this release, the default format for surfaces is changing from ASCII "


I'd remove "From this release" :)
And refactor the text a bit

xjules · 2026-02-13T13:36:38Z

What about storage migration? ..._file_format?

xjules

Nice one! 🚀

xjules · 2026-02-13T14:26:26Z

src/ert/config/surface_config.py

+ASCII_SURFACE_WARNING_MESSAGE = (
+    "The default format for surfaces is changing from ASCII "
+    "to binary, but Ert will temporarily still support ASCII surfaces. Binary "
+    "surfaces are recommended, as it is faster to write and read from runpath."


Let's add a sentence that the forward model creating the surfaces need to be changed too to export surfaces in binary format.

jonathan-eq · 2026-02-16T08:55:43Z

Added a note section in the docs, is this enough? @frode-aarstad

jonathan-eq · 2026-02-17T07:34:09Z

oyvindeide · 2026-02-17T09:57:46Z

src/ert/config/surface_config.py

            base_surface_path=base_surface,
            update=update_parameter,
        )
+        instance._file_format = file_format


Dont think this will work, because when we read from storage this will never be set, as an example, if you change:

ert/tests/ert/ui_tests/cli/test_parameter_sample_types.py

Line 44 in 4c1d474

base_surface.to_file("surf.irap", fformat="irap_ascii")

to irap_ascii I would expect this setting to be: ascii, but it is binary when we go to read the file from runpath.

No sure I understand. Base surface creates surf.irap that is binary and we still gonna read it from runpath as binary (by default). So binary all the way.

I think this needs more discussion @xjules @oyvindeide
If I set the default to ASCII, then old storages will be handled. If there is no key for file_format in parameters.json, then it will use ASCII and write file_format: ASCII to parameters.json and that will be picked up on the next reading. If there is a new field already in binary format, that will write file_format: BINARY to parameters.json, and that will also work downstream.

oyvindeide · 2026-02-17T09:59:40Z

src/ert/config/surface_config.py

@@ -59,6 +78,8 @@ class SurfaceConfig(ParameterConfig):
    output_file: Path
    base_surface_path: str



These file names should be mentioned explicitly in the warning message.

Before this commit, surfaces are only read by ert in ascii format. This commit changes the default to be binary, but falls back to ascii if it fails. It will also emit warnings if binary reading fails.

jonathan-eq added the release-notes:user-impact Automatically categorise as breaking for people using CLI/GUI label Feb 11, 2026

jonathan-eq self-assigned this Feb 11, 2026

jonathan-eq commented Feb 11, 2026

View reviewed changes

src/ert/config/surface_config.py Outdated Show resolved Hide resolved

jonathan-eq commented Feb 11, 2026

View reviewed changes

src/ert/config/surface_config.py Outdated Show resolved Hide resolved

jonathan-eq force-pushed the surface-warning branch 2 times, most recently from 0c2cb0c to 475a8c6 Compare February 11, 2026 09:11

jonathan-eq force-pushed the surface-warning branch 2 times, most recently from a074e70 to 14c8ab6 Compare February 11, 2026 11:12

jonathan-eq commented Feb 11, 2026

View reviewed changes

jonathan-eq force-pushed the surface-warning branch from 14c8ab6 to e7d45a3 Compare February 11, 2026 11:20

jonathan-eq commented Feb 11, 2026

View reviewed changes

src/ert/config/surface_config.py Outdated Show resolved Hide resolved

jonathan-eq force-pushed the surface-warning branch from b415594 to 4ff8eaf Compare February 11, 2026 12:10

frode-aarstad reviewed Feb 12, 2026

View reviewed changes

xjules reviewed Feb 13, 2026

View reviewed changes

xjules approved these changes Feb 13, 2026

View reviewed changes

xjules reviewed Feb 13, 2026

View reviewed changes

jonathan-eq force-pushed the surface-warning branch from 2a32793 to cd9d8a1 Compare February 16, 2026 08:56

jonathan-eq force-pushed the surface-warning branch from cd9d8a1 to d9b3d6b Compare February 17, 2026 07:43

oyvindeide reviewed Feb 17, 2026

View reviewed changes

jonathan-eq mentioned this pull request Feb 18, 2026

Add performance test for create_run_path for surface #12906

Merged

9 tasks

jonathan-eq force-pushed the surface-warning branch 2 times, most recently from 20437c4 to da961a5 Compare February 18, 2026 08:47

Migrate surfaces over to binary format

beb0e8e

Before this commit, surfaces are only read by ert in ascii format. This commit changes the default to be binary, but falls back to ascii if it fails. It will also emit warnings if binary reading fails.

jonathan-eq added 2 commits February 24, 2026 15:00

fixup! Default surface_config format to ascii to avoid storage migration

f07ec1a

fixup! Fix test

1a079ec

jonathan-eq force-pushed the surface-warning branch from da961a5 to 1a079ec Compare February 24, 2026 14:00

Add test that surface config writes same format as read from user

49e32d4

jonathan-eq force-pushed the surface-warning branch from ed4aff4 to 49e32d4 Compare February 25, 2026 09:44

		)


		@pytest.mark.parametrize("is_ascii_surface", [True, False])



		ASCII_SURFACE_WARNING_MESSAGE = (
		"From this release, the default format for surfaces is changing from ASCII "

		@@ -59,6 +78,8 @@ class SurfaceConfig(ParameterConfig):
		output_file: Path
		base_surface_path: str

Conversation

jonathan-eq commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

When applicable

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed-hq bot commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xjules commented Feb 13, 2026

Uh oh!

xjules left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathan-eq commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonathan-eq commented Feb 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jonathan-eq commented Feb 11, 2026 •

edited

Loading

codecov-commenter commented Feb 11, 2026 •

edited

Loading

codspeed-hq bot commented Feb 11, 2026 •

edited

Loading

jonathan-eq commented Feb 16, 2026 •

edited

Loading