Reproducibility fails with 0.4.0 build and the latest configuration #266

minghangli-uni · 2025-02-09T22:05:30Z

Below are the 5-day ocean.stats results highlighting the repro issue (an issue is raised here #265) when comparing the 0.3.0 and 0.4.0 builds with the 0.25 ryf configuration.

Using this existing configuration ACCESS-NRI/access-om3-configs@5842715 and the 0.3.0 build, ocean.stats shows,

Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err 
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 6.3048230599477494E-22, CFL  0.00000, SL  6.8348E-10, M 1.36283E+21, S 34.7205, T  3.6237, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    63,  693136.000,     0, En 1.1422421593679974E-03, CFL  0.12730, SL  6.8277E-10, M 1.36283E+21, S 34.7206, T  3.6237, Me -3.81E-19, Se -2.83E-17, Te  8.84E-19
   127,  693137.000,     0, En 1.1320364471729537E-03, CFL  0.11451, SL  6.8348E-10, M 1.36283E+21, S 34.7206, T  3.6236, Me  1.59E-19, Se  1.36E-17, Te -1.75E-18
   191,  693138.000,     0, En 1.1510039944012112E-03, CFL  0.09915, SL  6.8065E-10, M 1.36283E+21, S 34.7206, T  3.6236, Me  7.92E-20, Se  5.60E-19, Te  2.54E-18
   255,  693139.000,     0, En 1.1290348792421638E-03, CFL  0.09146, SL  6.8277E-10, M 1.36283E+21, S 34.7206, T  3.6237, Me  4.93E-19, Se  1.61E-17, Te -2.30E-18
   319,  693140.000,     0, En 1.1255847638941558E-03, CFL  0.09269, SL  6.8348E-10, M 1.36283E+21, S 34.7206, T  3.6237, Me  7.10E-20, Se  1.03E-17, Te  1.02E-18

Using this updated configuration ACCESS-NRI/access-om3-configs#169 with the 0.4.0 build, reproducibility fails despite having passed the existing repro test. For this specific case, 33 truncations occur on the 2nd day. But when using the latest topography and grid, the problem worsens, with over 20,000 truncation errors pop up ... (not shown here)

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err 
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 6.3048230599477494E-22, CFL  0.00000, SL  6.8348E-10, M 1.36283E+21, S 34.7205, T  3.6237, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    63,  693136.000,    33, En 1.1407486098043798E-03, CFL  0.12627, SL -3.4201E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me -9.42E-10, Se -4.89E-08, Te  5.68E-09
   127,  693137.000,     0, En 1.1294391817635600E-03, CFL  0.12267, SL -1.1492E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  6.25E-10, Se  1.30E-08, Te -3.45E-09
   191,  693138.000,     0, En 1.1501394646293486E-03, CFL  0.09017, SL  1.5319E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  7.38E-10, Se  2.36E-08, Te -2.95E-08
   255,  693139.000,     0, En 1.1299074397619121E-03, CFL  0.07864, SL  2.7025E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  3.22E-10, Se  1.11E-08, Te -4.45E-08
   319,  693140.000,     0, En 1.1271300577437651E-03, CFL  0.08655, SL  2.1515E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -1.52E-10, Se -4.01E-09, Te -1.84E-08

I was suspecting the issue might be related to the CICE grid, so I reverted to the old ice grid while keeping the 0.4.0 build. Then no truncations observed, but reproducibility failures persist...

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err 
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 6.3048230599477494E-22, CFL  0.00000, SL  6.8348E-10, M 1.36283E+21, S 34.7205, T  3.6237, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    63,  693136.000,     0, En 1.1422258589214400E-03, CFL  0.12731, SL -3.3210E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -9.15E-10, Se -4.80E-08, Te  5.75E-09
   127,  693137.000,     0, En 1.1319585613515698E-03, CFL  0.11500, SL -1.3026E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  5.56E-10, Se  1.08E-08, Te -4.95E-09
   191,  693138.000,     0, En 1.1508696289224552E-03, CFL  0.09993, SL  1.1858E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  6.85E-10, Se  2.18E-08, Te -3.17E-08
   255,  693139.000,     0, En 1.1288719261424492E-03, CFL  0.09160, SL  2.2845E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me  3.03E-10, Se  1.05E-08, Te -4.43E-08
   319,  693140.000,     0, En 1.1254764686950535E-03, CFL  0.09312, SL  1.9680E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -8.72E-11, Se -1.68E-09, Te -1.64E-08

So, it means the issue is likely not related to the cice grid alone, but also the latest 0.4.0 build? Or this is what 0.4.0 intentionally does? ping @anton-seaice @dougiesquire @chrisb13

The text was updated successfully, but these errors were encountered:

minghangli-uni · 2025-02-09T22:08:22Z

You may find the 3rd line in each ocean.stats is always the same.

anton-seaice · 2025-02-09T22:15:04Z

Do you have MOM_parameter_doc.all from 0.3.0 and 0.4.0, are their differences ?

minghangli-uni · 2025-02-09T22:39:30Z

Good question. No, they are not the same for some parameters. Some parameters that were default false in 0.3.0 are now default true!

I’ll check them in detail, ensure they are the same and run another test to see if they are consistent.

minghangli-uni · 2025-02-10T23:53:47Z

Below these parameters have been modified to ensure consistency with MOM parameters, but the bfb repro still fails. Here's where the differences are: https://github.com/minghangli-uni/access-om3-doc/blob/compare_0.3.0_0.4.0/tables/MOM_parameter_all_0.3.0-0.4.0/MOM6-CICE6-0.3.0-0.4.0_nml_diff.md

USE_POROUS_BARRIER = True       !   [Boolean] default = False
                                ! If true, use porous barrier to constrain the widths and face areas at the 
                                ! edges of the grid cells.

RESOLN_N2_FILTER_DEPTH = 2000.0   !   [m] default = -1.0
                                ! The depth below which N2 is monotonized to avoid stratification artifacts from
                                ! altering the equivalent barotropic mode structure.  This monotonzization is
                                ! disabled if this parameter is negative.
REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! This selects the remapping algorithm used in OM4 that does not use the full
                                ! reconstruction for the top- and lower-most sub-layers, but instead assumes
                                ! they are always vanished (untrue) and so just uses their edge values. We
                                ! recommend setting this option to false.

DIAG_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for diagnostics. See 
                                ! REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting this option to
                                ! false.

EBT_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for calculating EBT 
                                ! structure. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

HBD_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for horizontal boundary
                                ! diffusion. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

INTWAVE_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for calculating EBT 
                                ! structure. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

Z_INIT_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for initialization. See 
                                ! REMAPPING_USE_OM4_SUBCELLS for more details. We recommend setting this option
                                ! to false.

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err 
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 6.3048230599477494E-22, CFL  0.00000, SL  6.8348E-10, M 1.36283E+21, S 34.7205, T  3.6237, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    63,  693136.000,     0, En 1.1424681980749408E-03, CFL  0.12731, SL -3.3095E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -9.11E-10, Se -4.84E-08, Te  1.32E-09
   127,  693137.000,     0, En 1.1322062972728886E-03, CFL  0.11446, SL -1.0190E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  6.31E-10, Se  1.28E-08, Te -6.21E-09
   191,  693138.000,     0, En 1.1511417771868093E-03, CFL  0.09976, SL  1.3491E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  6.52E-10, Se  2.07E-08, Te -3.15E-08
   255,  693139.000,     0, En 1.1292283529569494E-03, CFL  0.09152, SL  2.4163E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me  2.94E-10, Se  1.00E-08, Te -4.56E-08
   319,  693140.000,     0, En 1.1258543366824954E-03, CFL  0.09256, SL  1.7924E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -1.72E-10, Se -4.66E-09, Te -1.75E-08

access-hive-bot · 2025-02-11T02:02:12Z

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/cosima-twg-announce/401/56

aekiss · 2025-02-11T22:40:58Z

No, they are not the same for some parameters. Some parameters that were default false in 0.3.0 are now default true!

If we stored up-to-date MOM_parameter_doc.* in the repo we'd catch these problems #117

dougiesquire · 2025-02-17T01:30:34Z

Below these parameters have been modified to ensure consistency with MOM parameters, but the bfb repro still fails. Here's where the differences are: https://github.com/minghangli-uni/access-om3-doc/blob/compare_0.3.0_0.4.0/tables/MOM_parameter_all_0.3.0-0.4.0/MOM6-CICE6-0.3.0-0.4.0_nml_diff.md

USE_POROUS_BARRIER = True       !   [Boolean] default = False
                                ! If true, use porous barrier to constrain the widths and face areas at the 
                                ! edges of the grid cells.

RESOLN_N2_FILTER_DEPTH = 2000.0   !   [m] default = -1.0
                                ! The depth below which N2 is monotonized to avoid stratification artifacts from
                                ! altering the equivalent barotropic mode structure.  This monotonzization is
                                ! disabled if this parameter is negative.
REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! This selects the remapping algorithm used in OM4 that does not use the full
                                ! reconstruction for the top- and lower-most sub-layers, but instead assumes
                                ! they are always vanished (untrue) and so just uses their edge values. We
                                ! recommend setting this option to false.

DIAG_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for diagnostics. See 
                                ! REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting this option to
                                ! false.

EBT_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for calculating EBT 
                                ! structure. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

HBD_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for horizontal boundary
                                ! diffusion. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

INTWAVE_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for calculating EBT 
                                ! structure. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

Z_INIT_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for initialization. See 
                                ! REMAPPING_USE_OM4_SUBCELLS for more details. We recommend setting this option
                                ! to false.

Noting for documentation purposes that this list is not correct.

The *_OM4_SUBCELLS parameters should be set to True to retain the original behaviour.
Changing USE_POROUS_BARRIER to True does nothing in our case (other than to slow the model down) since we don't configure any porous barriers. See Change default of USE_POROUS_BARRIER to false NOAA-GFDL/MOM6#735
Changing RESOLN_N2_FILTER_DEPTH to the previous default of 2000.0 actually changes answers. There was previously a bug that meant that RESOLN_N2_FILTER_DEPTH was actually being discarded. The new default disables monotonization, consistent with the previous buggy behaviour. See +(*)Fix wave_speed_init mono_N2_depth bug NOAA-GFDL/MOM6#413

There is actually only one change to the MOM6 parameters needed to reproduce MOM6 answers (see #270 (comment)):

USE_HUYNH_STENCIL_BUG = True    !   [Boolean] default = False
                                ! If true, use a stencil width of 2 in PPM:H3 tracer advection. This is
                                ! incorrect and will produce regressions in certain configurations, but may be
                                ! required to reproduce results in legacy simulations.

However, even with this parameter set, ACCESS-OM3 is not reproducible across 0.3.1 and 0.4.0 due to reproducibility-breaking changes in CICE, CMEPS and CDEPS - see ACCESS-NRI/access-om3-configs#173

anton-seaice · 2025-02-17T05:12:19Z

@minghangli-uni - did you look at reproducibility over restarts at all? I tried running the model-config-tests restart reproducability and it also failed:

[as2285@gadi-login-03 archive]$ cat MOM6-CICE6-B-test_restart_repro_2day/output000/ocean.stats
  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 3.1034198253312566E-23, CFL  0.00000, SL  1.5183E-10, M 1.36404E+21, S 34.7263, T  3.6362, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    23,  693136.000,     0, En 1.6467884517688178E-03, CFL  0.08144, SL -1.3225E-05, M 1.36404E+21, S 34.7263, T  3.6363, Me -3.63E-09, Se -1.67E-07, Te -3.44E-06
    47,  693137.000,     0, En 1.7271582761777793E-03, CFL  0.06106, SL -5.1177E-06, M 1.36404E+21, S 34.7263, T  3.6364, Me  2.22E-09, Se  1.40E-07, Te -2.71E-06
[as2285@gadi-login-03 archive]$ cat MOM6-CICE6-B-test_restart_repro_2x1day/output000/ocean.stats
  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 3.1034198253312566E-23, CFL  0.00000, SL  1.5183E-10, M 1.36404E+21, S 34.7263, T  3.6362, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    23,  693136.000,     0, En 1.6467884517688178E-03, CFL  0.08144, SL -1.3225E-05, M 1.36404E+21, S 34.7263, T  3.6363, Me -3.63E-09, Se -1.67E-07, Te -3.44E-06
[as2285@gadi-login-03 archive]$ cat MOM6-CICE6-B-test_restart_repro_2x1day/output001/ocean.stats
     0,  693136.000,     0, En 1.6468021784271944E-03, CFL  0.08144, SL -1.3225E-05, M 1.36404E+21, S 34.7263, T  3.6363, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    24,  693137.000,     0, En 1.7269455930905098E-03, CFL  0.06104, SL  1.5465E-05, M 1.36404E+21, S 34.7263, T  3.6364, Me  7.87E-09, Se  4.39E-07, Te -4.13E-07

Key points:

Did 2x1 day runs and 1 2 day run.

Results at the end of the first day seem the same

Some fields at the end of the second day are different. The error fields go back to 0 at the start of the second day in the 2x1 day run ? Is that expected ?

Still appears to not be very good at counting 24 hours !

minghangli-uni · 2025-02-17T05:43:16Z

Are these from 0.3.0 or 0.4.0? Restart repro works in 0.3.0 right? I haven’t checked 0.4.0 yet.

The error fields go back to 0 at the start of the second day in the 2x1 day run ? Is that expected ?

Yes, the error terms (Frac Mass Err, Salin Err, Temp Err) are recalculated independently for each run. They dont persist across runs cuz they are used to track errors within a single run but not across multiple runs. We should focus on terms beforehand.

There should be a separate restart repro check for each component. I dont think it comes from mom6 but might come from other components.

dougiesquire · 2025-02-17T06:23:50Z

@minghangli-uni - did you look at reproducibility over restarts at all? I tried running the model-config-tests restart reproducability and it also failed

Rats. I spy another rabbit hole

anton-seaice · 2025-02-18T23:18:52Z

I confirmed the result using the CI:

see ACCESS-NRI/access-om3-configs#177

The failure is comparing 2x1 day consecutive runs to 1x2 day run

There are three issues at a first glance:

comparing error fields in ocean.stats will always give different answers as they change with different start points in time (Frac Mass Err, Salin Err, Temp Err)
The two day run produces a ocean.stats with 3 lines in it, and the 1 day runs produces 2 ocean.stats with 2 lines in them, and we are trying to compare the all the lines. We should be trying to compare the last lines? (or creating the correct three lines from the 2x1 day runs).
En, CFL and SL are all different after two days.

dougiesquire · 2025-02-18T23:30:28Z

comparing error fields in ocean.stats will always give different answers as they change with different start points in time (Frac Mass Err, Salin Err, Temp Err)

The two day run produces a ocean.stats with 3 lines in it, and the 1 day runs produces 2 ocean.stats with 2 lines in them, and we are trying to compare the all the lines. We should be trying to compare the last lines? (or creating the correct three lines from the 2x1 day runs).

I think the implementation of the test should handle these fine since the test looks for the long-run checksums in the short-run checksums (not so say that things couldn't be improved for clarity)

En, CFL and SL are all different after two days.

Yeah, this is a problem...

aekiss · 2025-02-18T23:37:13Z

Why not compare the hashes of the restarts in the manifests? That should detect any failure of reproducibility, making tests of ocean.stats redundant (except that ocean.stats indicates repro breakdown with finer temporal resolution within a run rather than just detecting it at the end).

One wrinkle is that restart manifests track the initial rather than final state, so would need to update the manifests before comparing (does payu setup do that?)

anton-seaice · 2025-02-18T23:43:34Z

Why not compare the hashes of the restarts in the manifests? That should detect any failure of reproducibility, making tests of ocean.stats redundant (except that ocean.stats indicates repro breakdown with finer temporal resolution within a run rather than just detecting it at the end).

It seems like an overly stringent criteria to ? e.g. there's no particular reason for metadata in restart files to be the same for different runs

aekiss · 2025-02-18T23:59:07Z

But in practice (with om2 at least), I could compare restart md5 hashes to check reproducibility COSIMA/access-om2#266 so I guess there are no timestamps or whatnot in the restarts. Is that still the case with OM3 restarts?

(IIRC binhash can't be used because it includes the modification date https://payu.readthedocs.io/en/latest/manifests.html#manifest-contents, but md5 was ok)

dougiesquire · 2025-02-19T02:04:15Z

comparing error fields in ocean.stats will always give different answers as they change with different start points in time (Frac Mass Err, Salin Err, Temp Err)

The two day run produces a ocean.stats with 3 lines in it, and the 1 day runs produces 2 ocean.stats with 2 lines in them, and we are trying to compare the all the lines. We should be trying to compare the last lines? (or creating the correct three lines from the 2x1 day runs).

I think the implementation of the test should handle these fine since the test looks for the long-run checksums in the short-run checksums (not so say that things couldn't be improved for clarity)

As @anton-seaice pointed out to me at lunch, I'm of course wrong about the first point not being an issue. I'll update the tests to get this sorted.

dougiesquire · 2025-02-20T04:01:47Z

But in practice (with om2 at least), I could compare restart md5 hashes to check reproducibility COSIMA/access-om2#266 so I guess there are no timestamps or whatnot in the restarts. Is that still the case with OM3 restarts?

I think MOM6 restarts can be timestamped, but with the CESMCOUPLED cpp we don't use this feature. Restarts will also change for example if the mask table is changed, but our OM3 configs would abort in this case anyway because we use RESTART_CHECKSUMS_REQUIRED = True

anton-seaice · 2025-02-20T04:29:49Z

I also think we should be including experiment name, experiment id, author and contact in ALL output which would also break comparing md5 hashes of restart files.

Also there were issues reported with netcdf4 files produced by the ParallelIO library with identical contents having different checksums, I don't think they have been resolved (but nor are their many details around).

aekiss · 2025-02-21T04:15:57Z

I also think we should be including experiment name, experiment id, author and contact in ALL output which would also break comparing md5 hashes of restart files.

Or is breaking md5 a reason for not including all of this? (in restarts I mean, not outputs)

Also there were issues reported with netcdf4 files produced by the ParallelIO library with identical contents having different checksums, I don't think they have been resolved (but nor are their many details around).

Thanks, I hadn't been aware of that. Is it due to unpredictable chunk ordering or something?

minghangli-uni mentioned this issue Feb 9, 2025

Reading restart files issue using 0.4.0 build with asymmetric MOM6 #267

Closed

minghangli-uni mentioned this issue Feb 10, 2025

Update 0.25 deg MOM6 parameter ACCESS-NRI/access-om3-configs#73

Draft

minghangli-uni mentioned this issue Feb 11, 2025

Reproducibility issue with MOM6 standalone between 0.3.0 and 0.4.0 builds #270

Closed

This was referenced Feb 13, 2025

Debugging non-reproducibility of ACCESS-NRI 0.3.0 -> 0.4.0 ACCESS-NRI/ACCESS-OM3#46

Closed

Debugging non-reproducibility of ACCESS-NRI 0.3.1 -> 0.4.0 ACCESS-NRI/ACCESS-OM3#47

Draft

This was referenced Feb 19, 2025

Reproducibility tests for determinism from restart (as well as cold start) #277

Open

Use restart md5 hashes for reproducibility tests #278

Open

anton-seaice mentioned this issue Feb 20, 2025

Update wav- configs #283

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reproducibility fails with 0.4.0 build and the latest configuration #266

Reproducibility fails with 0.4.0 build and the latest configuration #266

minghangli-uni commented Feb 9, 2025

minghangli-uni commented Feb 9, 2025

anton-seaice commented Feb 9, 2025

minghangli-uni commented Feb 9, 2025

minghangli-uni commented Feb 10, 2025

access-hive-bot commented Feb 11, 2025

aekiss commented Feb 11, 2025

dougiesquire commented Feb 17, 2025

anton-seaice commented Feb 17, 2025

minghangli-uni commented Feb 17, 2025

dougiesquire commented Feb 17, 2025

anton-seaice commented Feb 18, 2025 •

edited

Loading

dougiesquire commented Feb 18, 2025 •

edited

Loading

aekiss commented Feb 18, 2025 •

edited

Loading

anton-seaice commented Feb 18, 2025

aekiss commented Feb 18, 2025 •

edited

Loading

dougiesquire commented Feb 19, 2025

dougiesquire commented Feb 20, 2025

anton-seaice commented Feb 20, 2025

aekiss commented Feb 21, 2025

Reproducibility fails with 0.4.0 build and the latest configuration #266

Reproducibility fails with 0.4.0 build and the latest configuration #266

Comments

minghangli-uni commented Feb 9, 2025

minghangli-uni commented Feb 9, 2025

anton-seaice commented Feb 9, 2025

minghangli-uni commented Feb 9, 2025

minghangli-uni commented Feb 10, 2025

access-hive-bot commented Feb 11, 2025

aekiss commented Feb 11, 2025

dougiesquire commented Feb 17, 2025

anton-seaice commented Feb 17, 2025

minghangli-uni commented Feb 17, 2025

dougiesquire commented Feb 17, 2025

anton-seaice commented Feb 18, 2025 • edited Loading

dougiesquire commented Feb 18, 2025 • edited Loading

aekiss commented Feb 18, 2025 • edited Loading

anton-seaice commented Feb 18, 2025

aekiss commented Feb 18, 2025 • edited Loading

dougiesquire commented Feb 19, 2025

dougiesquire commented Feb 20, 2025

anton-seaice commented Feb 20, 2025

aekiss commented Feb 21, 2025

anton-seaice commented Feb 18, 2025 •

edited

Loading

dougiesquire commented Feb 18, 2025 •

edited

Loading

aekiss commented Feb 18, 2025 •

edited

Loading

aekiss commented Feb 18, 2025 •

edited

Loading