🐛[BUG]: Disagreement on sea surface temperature between DLESyM and published results

[chdamianos]

Version

0.7.0

On which installation method(s) does this occur?

pip

Describe the issue

I compared the forecasts produced by the DLESyM implemented in Earth2Studio 0.7.0 and the forecasts published from the DLESyM paper on DLESyM’s GitHub repository in the 100yr_hpx64_6month_rolling_fcst_monthly_averages_sst.nc file. I have found that the El Niño–Southern Oscillation (ENSO) region SST forecasts from the paper to be significantly different from the ones I’m getting from the Earth2Studio version of DLESyM.

Also, I found that the forecasts from the paper agree better with the ERA5 ENSO SST data than the forecasts I get from the Earth2Studio version of DLESyM.

I know that the Earth2Studio DLESyM version and the one published in the DLESyM paper are not the same since the Earth2Studio version doesn’t make use of "outgoing longwave radiation" (OLR), as mentioned in this issue. But I’m not sure if this difference is the cause of the disagreement when it comes to forecasting SST from the ENSO region or if there’s an issue with the Earth2Studio implementation.

I’m sharing a code snippet and the files to reproduce my findings. It’s a 5-year forecast of the Earth2Studio version of DLESyM starting on 2016-12-15 averaged monthly to match the data published in the 100yr_hpx64_6month_rolling_fcst_monthly_averages_sst.nc file of the DLESyM repo.

I have also extracted the ENSO region ERA5 SST averages using Earth2Studio’s WB2ERA5 data source to compare with both DLESyM forecasts.

I have compared both forecasts with the ERA5 SST average timeseries in the plot below and also calculated the correlation between the timeseries:

• The correlation between the ERA5 SST timeseries and the Earth2Studio DLESyM forecast is 0.05
• Whereas the correlation between the ERA5 SST timeseries and the forecast from the DLESyM GitHub repository is 0.54

As you can see from the plot the Earth2Studio DLESyM version and the DLESyM paper forecasts don’t match that well.

From the correlation values (0.54 vs 0.05) the DLESyM in paper has a much better match with the ERA5 SST timeseries.

You can recreate the plot below by running the dlesym_minimal_example.py attached. The script was run with earth2studio version 0.7.0. I’m also attaching the ERA5 SST averages (wb2era5.csv). The 100yr_hpx64_6month_rolling_fcst_monthly_averages_sst.nc file can be downloaded from the DLESyM repo.

The labels of the plot are assigned as:

• 100yr_hpx64_6month_rolling_fcst_monthly_averages_sst.nc: Baseline
• ERA5 SST averages: WB2ERA5
• DLESyM version from Earth2Studio: Earth2Studio

[pzharrington]

Hi @chdamianos -- thank you for this very thorough analysis and reproducer. You've already fished out the reason for this discrepancy:

the Earth2Studio DLESyM version and the one published in the DLESyM paper are not the same since the Earth2Studio version doesn’t make use of "outgoing longwave radiation" (OLR)

The earth2studio DLESyM has different models in both the atmosphere and ocean components than those presented in the original paper, and was designed more with subseasonal-to-seasonal (S2S) forecasting in mind. The DLESyM team does not expect the current set of checkpoints earth2studio to fully reproduce all the long-term/climate-timescale metrics from the original paper. They are working on developing a new set of models that will try to capture both the long-term rollout behavior as well as near-term S2S skill with the same checkpoints, and those will be released via earth2studio when ready.

In the meantime, we can also look into the viability of also releasing the original set of checkpoints as an additional model package, though that may not be fully possible for various reasons. I will update this issue when I have more news. Thanks again for investigating and raising!

[chdamianos]

Thanks @pzharrington for your prompt reply! I'm looking forward to your updates.