EAMxx variables #880

chengzhuzhang · 2024-10-29T17:10:23Z

Description

This PR replaces More EAMXX var support #849
This enhancement will be merged in a new e3sm_diags code base after cdat-migration-fy24 branch is merged.

Reference:
The mapping of new variables is based on https://acme-climate.atlassian.net/wiki/spaces/EAMXX/pages/4535976058/Output+Standard+Names put together by @AaronDonahue
The decadal output outlined by @brhillman: https://github.com/E3SM-Project/eamxx-scripts/pull/180/files#diff-1646ba1e37781387625d2ce585aad9ef7f5b6407616300838c7aecd44c67df7e

Closes [Bug]: CDAT migration: various KeyError for longitude bounds #884

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules

If applicable:

New and existing unit tests pass with my changes (locally and CI/CD build)
I have added tests that prove my fix is effective or that my feature works
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

tomvothecoder · 2024-11-04T23:55:43Z

e3sm_diags/driver/utils/dataset_xr.py

            ds_climo = climo(ds, self.var, season).to_dataset()
+            ds_climo = ds_climo.bounds.add_missing_bounds(axes=["X", "Y"])


This should close #884. Please verify the fix, thanks.

Indeed, this fixed the problem! Thank you!

chengzhuzhang · 2024-11-05T19:20:30Z

@tomvothecoder I completed 2D and 3D variables derivation, excepted for COSP related output.
The only thing left here is that we need to support lowercase landfrac/ocnfrac that from EAMxx. The code block is as follows:

e3sm_diags/e3sm_diags/driver/__init__.py

Lines 7 to 12 in 43d2df7

    
           LAND_OCEAN_MASK_PATH = os.path.join(INSTALL_PATH, "acme_ne30_ocean_land_mask.nc") 
        
           # The keys for the land and ocean fraction variables in the 
        
           # `LAND_OCEAN_MASK_PATH` file. 
        
           LAND_FRAC_KEY = "LANDFRAC" 
        
           OCEAN_FRAC_KEY = "OCNFRAC"

I'm now sure how to provide a clean way to accommodate lower case variable names..

tomvothecoder · 2024-11-05T23:42:42Z

The only thing left here is that we need to support lowercase landfrac/ocnfrac that from EAMxx.

I just pushed 46a5dad to add support for more land/ocean var keys. Let me know your thoughts.

tomvothecoder · 2024-11-06T17:56:57Z

Pushed ef261b8 (#880) to make land sea mask methods a bit cleaner. Should be good to go.

chengzhuzhang · 2024-11-07T00:26:36Z

e3sm_diags/driver/utils/regrid.py

@@ -243,7 +242,7 @@ def _apply_land_sea_mask(
    ds_new = ds.copy()
    ds_new = _drop_unused_ilev_axis(ds)
    output_grid = ds_new.regridder.grid
-    mask_var_key = _get_region_mask_var_key(region)
+    mask_var_key = _get_region_mask_var_key(ds_mask, region)


I ran into error below for a special usecase when the region is land_60S90N.

raise ValueError(f"Only land and ocean regions are supported, not '{region}'.") ValueError: Only land and ocean regions are supported, not 'land_60S90N'.

It looks like we should have a way to distinguish land/ocean mask only and the other type of special regions.

Commit 542b88b should address this.

tomvothecoder · 2024-11-08T20:32:50Z

e3sm_diags/driver/utils/dataset_xr.py

-        ds_sub = self._subset_vars_and_load(ds, var)
-
-        time_slice = self._get_time_slice(ds_sub)
-        ds_sub = ds_sub.sel(time=time_slice).squeeze()
+        time_slice = self._get_time_slice(ds)
+        ds_sub = ds.sel(time=time_slice).squeeze()

        if self.is_sub_monthly:
            ds_sub = self._exclude_sub_monthly_coord_spanning_year(ds_sub)

+        ds_sub = self._subset_vars_and_load(ds_sub, var)
+
        return ds_sub


This should close #892. I was able to run the U variable. Can you please confirm the fix too @chengzhuzhang?

My output:

2024-11-08 12:52:52,411 [INFO]: e3sm_diags_driver.py(_save_env_yml:58) >> Saved environment yml file to: [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/prov/environment.yml](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/prov/environment.yml) 2024-11-08 12:52:52,413 [INFO]: e3sm_diags_driver.py(_save_parameter_files:69) >> Saved command used to: [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/prov/cmd_used.txt](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/prov/cmd_used.txt) 2024-11-08 12:52:52,414 [INFO]: e3sm_diags_driver.py(_save_python_script:133) >> Saved Python script to: [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/prov/ipykernel_launcher.py](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/prov/ipykernel_launcher.py) 2024-11-08 12:52:52,976 [INFO]: lat_lon_driver.py(run_diag:69) >> Variable: U 2024-11-08 12:53:12,435 [INFO]: lat_lon_driver.py(_run_diags_3d:396) >> Selected pressure level(s): [850.0] 2024-11-08 12:53:13,554 [INFO]: regrid.py(subset_and_align_datasets:70) >> Selected region: global 2024-11-08 12:53:21,676 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.json](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.json) 2024-11-08 12:53:28,892 [INFO]: utils.py(_save_plot:91) >> Plot saved in: [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.png](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.png) 2024-11-08 12:53:28,892 [INFO]: utils.py(_save_plot:91) >> Plot saved in: [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.png](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.png) 2024-11-08 12:53:28,895 [INFO]: main.py(create_viewer:132) >> lat_lon [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/viewer](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/viewer) 2024-11-08 12:53:32,950 [INFO]: main.py(create_viewer:135) >> [('Latitude-Longitude contour maps', 'lat_lon/index.html'), ('Table', 'table/index.html'), ('Taylor Diagram', 'taylor/index.html'), ('CMIP6 Comparison', 'cmip6/index.html')] 2024-11-08 12:53:32,956 [INFO]: e3sm_diags_driver.py(main:392) >> Viewer HTML generated at [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/viewer/index.html](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/viewer/index.html) 2024-11-08 12:53:32,976 [INFO]: logger.py(move_log_to_prov_dir:106) >> Log file saved in [/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/prov/e3sm_diags_run.log](https://vscode-remote+ssh-002dremote-002bperlmutter.vscode-resource.vscode-cdn.net/global/cfs/cdirs/e3sm/www/cdat-migration-fy24/eamxx_decadal_1996_1031_edv3/prov/e3sm_diags_run.log)

@tomvothecoder When testing after re-basing, issue #892 came back. Now I'm having 30 mins to complete this run:

(/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv3) chengzhu@nid004205:~/eamxx_diags/run_script> python run_e3sm_diags_1996.py -d U_lat_lon.cfg 2024-12-12 15:23:42,166 [INFO]: e3sm_diags_driver.py(_save_env_yml:58) >> Saved environment yml file to: /global/cfs/cdirs/e3sm/www/zhang40/tests/eamxx/eamxx_decadal_1996_1212_edv3_U/prov/environment.yml 2024-12-12 15:23:42,172 [INFO]: e3sm_diags_driver.py(_save_parameter_files:69) >> Saved command used to: /global/cfs/cdirs/e3sm/www/zhang40/tests/eamxx/eamxx_decadal_1996_1212_edv3_U/prov/cmd_used.txt 2024-12-12 15:23:42,184 [INFO]: e3sm_diags_driver.py(_save_parameter_files:99) >> Saved cfg file to: /global/cfs/cdirs/e3sm/www/zhang40/tests/eamxx/eamxx_decadal_1996_1212_edv3_U/prov/U_lat_lon.cfg 2024-12-12 15:23:42,191 [INFO]: e3sm_diags_driver.py(_save_python_script:133) >> Saved Python script to: /global/cfs/cdirs/e3sm/www/zhang40/tests/eamxx/eamxx_decadal_1996_1212_edv3_U/prov/run_e3sm_diags_1996.py 2024-12-12 15:23:53,074 [INFO]: lat_lon_driver.py(run_diag:69) >> Variable: U 2024-12-12 15:53:23,937 [INFO]: lat_lon_driver.py(_run_diags_3d:396) >> Selected pressure level(s): [850.0] 2024-12-12 15:53:27,832 [INFO]: regrid.py(subset_and_align_datasets:70) >> Selected region: global 2024-12-12 15:53:40,129 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/zhang40/tests/eamxx/eamxx_decadal_1996_1212_edv3_U/lat_lon/ERA5/ERA5-U-850-ANN-global.json 2024-12-12 15:53:48,999 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/zhang40/tests/eamxx/eamxx_decadal_1996_1212_edv3_U/lat_lon/ERA5/ERA5-U-850-ANN-global.png 2024-12-12 15:53:49,007 [INFO]: main.py(create_viewer:132) >> lat_lon /global/cfs/cdirs/e3sm/www/zhang40/tests/eamxx/eamxx_decadal_1996_1212_edv3_U/viewer 2024-12-12 15:54:02,811 [INFO]: main.py(create_viewer:135) >> [('Latitude-Longitude contour maps', 'lat_lon/index.html'), ('Table', 'table/index.html'), ('Taylor Diagram', 'taylor/index.html'), ('CMIP6 Comparison', 'cmip6/index.html')] 2024-12-12 15:54:02,847 [INFO]: e3sm_diags_driver.py(main:392) >> Viewer HTML generated at /global/cfs/cdirs/e3sm/www/zhang40/tests/eamxx/eamxx_decadal_1996_1212_edv3_U/viewer/index.html

I think the code diff look okay though, the code change needed is carried over, but not sure why the performance issue returned.

Thanks for re-running the script. I will debug the performance bottleneck when I'm back Mon.

This change should be on main anyways. I am going to port it over to #907.

tomvothecoder · 2024-11-08T20:33:46Z

I'm getting a separate error about circular imports. Do you see this too?

Not sure how this was introduced, but it should be addressed:

2024-11-08 12:27:54,820 [ERROR]: core_parameter.py(_run_diag:343) >> Error in e3sm_diags.driver.lat_lon_driver
Traceback (most recent call last):
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/parameter/core_parameter.py", line 340, in _run_diag
    single_result = module.run_diag(self)
AttributeError: partially initialized module 'e3sm_diags.driver.lat_lon_driver' has no attribute 'run_diag' (most likely due to a circular import)
2024-11-08 12:27:54,821 [ERROR]: run.py(run_diags:91) >> Error traceback:
Traceback (most recent call last):
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/run.py", line 89, in run_diags
    params_results = main(params)
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/e3sm_diags_driver.py", line 373, in main
    parameters_results = _run_serially(parameters)
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/e3sm_diags_driver.py", line 271, in _run_serially
    nested_results.append(parameter._run_diag())
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/parameter/core_parameter.py", line 333, in _run_diag
    module = importlib.import_module(mod_str)
  File "/global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_892/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/driver/zonal_mean_xy_driver.py", line 17, in <module>
    from e3sm_diags.metrics.metrics import spatial_avg
ImportError: cannot import name 'spatial_avg' from partially initialized module 'e3sm_diags.metrics.metrics' (most likely due to a circular import) (/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/metrics/metrics.py)
2024-11-08 12:27:54,824 [INFO]: logger.py(move_log_to_prov_dir:106) >> Log file saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/877-attr-err/eamxx_decadal_1996_1107_edv3/prov/e3sm_diags_run.log
2024-11-08 12:27:55,985 [INFO]: lat_lon_driver.py(run_diag:69) >> Variable: U
Value(False)
2024-11-08 12:32:27,811 [INFO]: lat_lon_driver.py(_run_diags_3d:396) >> Selected pressure level(s): [850.0]
2024-11-08 12:32:29,678 [INFO]: regrid.py(subset_and_align_datasets:70) >> Selected region: global
2024-11-08 12:32:39,801 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/877-attr-err/eamxx_decadal_1996_1107_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.json
2024-11-08 12:32:54,463 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/877-attr-err/eamxx_decadal_1996_1107_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.png
2024-11-08 12:32:54,463 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/877-attr-err/eamxx_decadal_1996_1107_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.png

tomvothecoder · 2024-11-08T20:53:37Z

I'm getting a separate error about circular imports. Do you see this too?

Not sure how this was introduced, but it should be addressed:

2024-11-08 12:27:54,820 [ERROR]: core_parameter.py(_run_diag:343) >> Error in e3sm_diags.driver.lat_lon_driver
Traceback (most recent call last):
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/parameter/core_parameter.py", line 340, in _run_diag
    single_result = module.run_diag(self)
AttributeError: partially initialized module 'e3sm_diags.driver.lat_lon_driver' has no attribute 'run_diag' (most likely due to a circular import)
2024-11-08 12:27:54,821 [ERROR]: run.py(run_diags:91) >> Error traceback:
Traceback (most recent call last):
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/run.py", line 89, in run_diags
    params_results = main(params)
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/e3sm_diags_driver.py", line 373, in main
    parameters_results = _run_serially(parameters)
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/e3sm_diags_driver.py", line 271, in _run_serially
    nested_results.append(parameter._run_diag())
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/parameter/core_parameter.py", line 333, in _run_diag
    module = importlib.import_module(mod_str)
  File "/global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_892/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/driver/zonal_mean_xy_driver.py", line 17, in <module>
    from e3sm_diags.metrics.metrics import spatial_avg
ImportError: cannot import name 'spatial_avg' from partially initialized module 'e3sm_diags.metrics.metrics' (most likely due to a circular import) (/global/u2/v/vo13/E3SM-Project/e3sm_diags/e3sm_diags/metrics/metrics.py)
2024-11-08 12:27:54,824 [INFO]: logger.py(move_log_to_prov_dir:106) >> Log file saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/877-attr-err/eamxx_decadal_1996_1107_edv3/prov/e3sm_diags_run.log
2024-11-08 12:27:55,985 [INFO]: lat_lon_driver.py(run_diag:69) >> Variable: U
Value(False)
2024-11-08 12:32:27,811 [INFO]: lat_lon_driver.py(_run_diags_3d:396) >> Selected pressure level(s): [850.0]
2024-11-08 12:32:29,678 [INFO]: regrid.py(subset_and_align_datasets:70) >> Selected region: global
2024-11-08 12:32:39,801 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/877-attr-err/eamxx_decadal_1996_1107_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.json
2024-11-08 12:32:54,463 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/877-attr-err/eamxx_decadal_1996_1107_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.png
2024-11-08 12:32:54,463 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/877-attr-err/eamxx_decadal_1996_1107_edv3/lat_lon/ERA5/ERA5-U-850-ANN-global.png

No longer appearing after running make install again. Good to go here.

chengzhuzhang · 2024-11-08T20:58:08Z

Commit #892 worked well! It took about 1 min to finish the 3d variableU run, which is comparable to what cdat does. Thank you for a quick fix! @tomvothecoder

chengzhuzhang · 2024-11-08T23:01:26Z

I found another issue that derived variable is not working for time-series files. Example .cfg

[#]
sets = ["lat_lon"]
case_id = "ERA5"
variables = ["QREFHT"]
ref_name = "ERA5_ext"
reference_name = "ERA5 Reanalysis"
seasons = ["ANN", "01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "DJF", "MAM", "JJA", "SON"]
contour_levels = [0.2, 0.5, 1, 2.5, 5, 7.5, 10, 12.5, 15, 17.5]
diff_levels = [-5, -4, -3, -2, -1, -0.25, 0.25, 1, 2, 3, 4, 5]

The input files d2m and sp files (which are ERA5 variables used to derive QREFHT ) are available in time-series/EAM5_ext directory. But the program is trying to look for:
/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/time-series/ERA5_ext/QREFHT_.{13}.nc

tomvothecoder · 2024-11-08T23:56:36Z

I found another issue that derived variable is not working for time-series files. Example .cfg
[#]
sets = ["lat_lon"]
case_id = "ERA5"
variables = ["QREFHT"]
ref_name = "ERA5_ext"
reference_name = "ERA5 Reanalysis"
seasons = ["ANN", "01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "DJF", "MAM", "JJA", "SON"]
contour_levels = [0.2, 0.5, 1, 2.5, 5, 7.5, 10, 12.5, 15, 17.5]
diff_levels = [-5, -4, -3, -2, -1, -0.25, 0.25, 1, 2, 3, 4, 5]
The input files d2m and sp files (which are ERA5 variables used to derive QREFHT ) are available in time-series/EAM5_ext directory. But the program is trying to look for: /global/cfs/cdirs/e3sm/diagnostics/observations/Atm/time-series/ERA5_ext/QREFHT_.{13}.nc

You may need to step-through the loop that attempts to derive QREFHT until it hits d2m and sp. I walked through the code and it tries to match on these two filepath patterns, which look correct.

/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/time-series/ERA5_ext/d2m_.{13}.nc
/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/time-series/ERA5_ext/sp_.{13}.nc.

Possible issue

The ERA5_ext sub-directory doesn't look like it exists under the root directory (/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/time-series).

(e3sm_diags_dev_892) vo13@login08:.../Atm/time-series$ pwd
/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/time-series
(e3sm_diags_dev_892) vo13@login08:.../Atm/time-series$ ls ERA5_ext
ls: cannot access 'ERA5_ext': No such file or directory
(e3sm_diags_dev_892) vo13@login08:.../Atm/time-series$

chengzhuzhang · 2024-11-08T23:59:46Z

@tomvothecoder thank you for looking into this! I'm actually stepping into the code, and it does look like the logic should be correct. Yes, the files are mis-placed to EAR5, but should be in a seperate directory ERA5-ext. Let me fix the data and try again.

chengzhuzhang · 2024-11-09T00:43:47Z

I can confirm that this is a data problem. The problem described in #880 (comment) is resolved with time-series files placed in correct directory ERA5-ext. The data fix is ready on lcrc and perlmutter.

tomvothecoder

Hi @chengzhuzhang, this PR looks good to me. I had some minor comments/questions. Thanks for working on this PR!

tomvothecoder · 2024-11-11T19:41:28Z

e3sm_diags/derivations/derivations.py

+        ("precip_liq_surf_mass_flux", "precip_ice_surf_mass_flux"): prect,  # EAMxx
+        ("precip_total_surf_mass_flux",): lambda pr: convert_units(
+            rename(pr), target_units="mm/day"
+        ),  # EAMxx


Should we considering separating EAMxx variables out to another derivations dictionary so we don't need to comment # EAMxx?

This can be done through #716.

let's keep the comments for now and we can consider separation in the future. We probably want to separate variables from multiple obs sources as well.

e3sm_diags/derivations/derivations.py

e3sm_diags/driver/utils/dataset_xr.py

tomvothecoder · 2024-11-11T19:49:36Z

I think this branch needs to be rebased on the latest cdat-migration-fy24. Let me do this and then we can merge whenever you're ready.

chengzhuzhang · 2024-11-11T19:51:20Z

I think this branch needs to be rebased on the latest cdat-migration-fy24. Let me do this and then we can merge whenever you're ready.

Thank you for the review and rebasing @tomvothecoder. I will tag EAMxx developers for a review before merging.

tomvothecoder · 2024-11-11T19:59:32Z

Just rebased, should be good to go for further review.

chengzhuzhang · 2024-11-11T22:39:37Z

Hi @PeterCaldwell @brhillman @crterai @AaronDonahue:

This PR added support for all EAMxx output variables (except for those COSP related). This takes a little longer because, this update is based on the brand new e3sm_diags that is just migrated over to use xarray/xcdat to replace cdat (kudos to @tomvothecoder).

Here is an example e3sm_diags run based on the 1996ish EAMxx decadal run that Ben provided.

The workflow to generate this run is to first run ncclimo to generate the regridded climatology file ; and then run the e3sm_diags run script.

Example of the nco script as below. Thanks to @czender, the two nco steps listed below can be simplified with just using one ncclimo command line with latest nco release. The improvement will be available through the next e3sm-unified release scheduled in February.

#!/bin/bash                                 

source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

drc_in=/global/cfs/cdirs/e3sm/chengzhu/eamxx/run
drc_out=/global/cfs/cdirs/e3sm/chengzhu/eamxx/post/data
caseid=output.scream.decadal.monthlyAVG_ne30pg2.AVERAGE.nmonths_x1

# spoofed climatology files with data from 1995-09 to 1996-08

# create climatology files
cd ${drc_in};ls ${caseid}*1996-0[1-8]*.nc ${caseid}*1995-09*.nc ${caseid}*1995-1[0-2]*.nc | ncclimo -P eamxx --fml_nm=eamxx_decadal --yr_srt=1996 --yr_end=1996 --drc_out=$drc_out


map=/global/cfs/projectdirs/e3sm/zender/maps/map_ne30pg2_to_cmip6_180x360_traave.20231201.nc
# remaping climo files to regular lat-lon
cd $drc_out;ls *.nc | ncremap -P eamxx --prm_opt=time,lwband,swband,ilev,lev,plev,cosp_tau,cosp_cth,cosp_prs,dim2,ncol --map=${map} --drc_out=${drc_out}/rgr

exit

It would be great if you could review the code change to verify the variable derivations are correct. Next we will work on better support on arbitrary length runs; COSP histogram and other variability sets (if given longer simulation). Any feedback on capabilities and priorities are welcome!

Thanks,
Jill

- The time slice should occur before loading the dataset into memory, otherwise the entire time series will be loaded which can be a large dataset Add testing scripts Remove accidental rebase changes

tomvothecoder · 2025-01-15T21:37:06Z

01/15/25 final checklist before merging:

Rebase on latest main with PR Address diffs v2.12.1 to v3 #907 merged
Code review on GitHub
Complete run on Perlmutter (currently down on 1/15) -- IN PROGRESS
Regression testing -- confirm script completes within 2 hours like before
Merge this PR!

e3sm_diags/driver/utils/dataset_xr.py

tomvothecoder · 2025-01-16T21:16:17Z

I re-ran examples/e3sm_diags_for_eamxx/run_e3sm_diags_1996.py with this branch rebased on the latest main including #907. It ran successfully and all of the results look like they were produced: https://portal.nersc.gov/project/e3sm/vo13/tests/eamxx/eamxx_decadal_1996_1212_edv3/viewer/

I will merge this branch now!

tomvothecoder changed the base branch from main to cdat-migration-fy24 October 29, 2024 17:50

tomvothecoder force-pushed the cdat-migration-fy24 branch 2 times, most recently from d5a1aad to 7550b3d Compare October 29, 2024 21:18

tomvothecoder force-pushed the eamxx_1024 branch 2 times, most recently from 2c55d9e to 469cebb Compare November 4, 2024 19:47

tomvothecoder reviewed Nov 4, 2024

View reviewed changes

tomvothecoder mentioned this pull request Nov 4, 2024

[Bug]: CDAT migration: various KeyError for longitude bounds #884

Closed

chengzhuzhang assigned chengzhuzhang and tomvothecoder Nov 5, 2024

chengzhuzhang commented Nov 7, 2024

View reviewed changes

tomvothecoder reviewed Nov 8, 2024

View reviewed changes

chengzhuzhang marked this pull request as ready for review November 9, 2024 00:48

tomvothecoder approved these changes Nov 11, 2024

View reviewed changes

tomvothecoder added the enhancement label Nov 11, 2024

tomvothecoder added this to the FY25 Q1 (10/1/24 - 12/31/24) milestone Nov 11, 2024

tomvothecoder force-pushed the eamxx_1024 branch from 47d0a0a to d94a66d Compare November 11, 2024 19:55

chengzhuzhang and others added 19 commits January 15, 2025 15:06

more informative log

a346098

Remove inadvertant rebase diffs

3ab50ae

Update IOError

c3dbdd6

Add bounds after calculating climo for time series

bfc51a1

Fix unit test

f49f606

fixing radiation fluxes and units

849ea26

Add support for more land ocean var keys

f323ecd

Refactor land sea mask methods

b34caaf

fixing for regions i.e. land_60N60S

5115c6a

Update order of time series subsetting to improve performance

35fbf20

- The time slice should occur before loading the dataset into memory, otherwise the entire time series will be loaded which can be a large dataset Add testing scripts Remove accidental rebase changes

Revert accidental rebase changes

4ddf616

Add run script for bottleneck

64629a0

Add FIXME comments for performance bottleneck

0009dea

Add debug_ref_u script

9d00cbf

Add debug scripts

3ba6ec5

add run scripts examples

e52c7da

address review; clean up

02b96e0

Update README.md

1a9152d

Update e3sm_diags/driver/lat_lon_driver.py

3bbc697

tomvothecoder force-pushed the eamxx_1024 branch from 181564e to 3bbc697 Compare January 15, 2025 21:07

tomvothecoder changed the base branch from cdat-migration-fy24 to main January 15, 2025 21:17

tomvothecoder reviewed Jan 15, 2025

View reviewed changes

e3sm_diags/driver/utils/dataset_xr.py Outdated Show resolved Hide resolved

e3sm_diags/driver/utils/dataset_xr.py Outdated Show resolved Hide resolved

e3sm_diags/driver/utils/dataset_xr.py Outdated Show resolved Hide resolved

tomvothecoder added 2 commits January 15, 2025 14:41

Apply suggestions from code review

c70efd1

Fix pre-commit issues

8609b10

tomvothecoder modified the milestones: FY25 Q1 (10/1/24 - 12/31/24), FY25 Q2 (01/01/25 - 03/31/25) Jan 16, 2025

tomvothecoder mentioned this pull request Jan 16, 2025

Replace deprecated SciPy cwt with PyWavelets cwt #913

Merged

9 tasks

tomvothecoder merged commit d01fd7a into main Jan 16, 2025
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EAMxx variables #880

EAMxx variables #880

chengzhuzhang commented Oct 29, 2024 •

edited

Loading

tomvothecoder Nov 4, 2024

chengzhuzhang Nov 5, 2024

chengzhuzhang commented Nov 5, 2024 •

edited

Loading

tomvothecoder commented Nov 5, 2024

tomvothecoder commented Nov 6, 2024

chengzhuzhang Nov 7, 2024

chengzhuzhang Nov 8, 2024

tomvothecoder Nov 8, 2024

tomvothecoder Nov 8, 2024

chengzhuzhang Dec 13, 2024 •

edited

Loading

tomvothecoder Dec 13, 2024

tomvothecoder Dec 17, 2024

tomvothecoder commented Nov 8, 2024

tomvothecoder commented Nov 8, 2024

chengzhuzhang commented Nov 8, 2024 •

edited

Loading

chengzhuzhang commented Nov 8, 2024 •

edited

Loading

tomvothecoder commented Nov 8, 2024

chengzhuzhang commented Nov 8, 2024

chengzhuzhang commented Nov 9, 2024 •

edited

Loading

tomvothecoder left a comment

tomvothecoder Nov 11, 2024

chengzhuzhang Dec 20, 2024

tomvothecoder commented Nov 11, 2024

chengzhuzhang commented Nov 11, 2024 •

edited

Loading

tomvothecoder commented Nov 11, 2024 •

edited

Loading

chengzhuzhang commented Nov 11, 2024

tomvothecoder commented Jan 15, 2025 •

edited

Loading

tomvothecoder commented Jan 16, 2025

		ds_climo = climo(ds, self.var, season).to_dataset()
		ds_climo = ds_climo.bounds.add_missing_bounds(axes=["X", "Y"])

EAMxx variables #880

EAMxx variables #880

Conversation

chengzhuzhang commented Oct 29, 2024 • edited Loading

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chengzhuzhang commented Nov 5, 2024 • edited Loading

tomvothecoder commented Nov 5, 2024

tomvothecoder commented Nov 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chengzhuzhang Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomvothecoder commented Nov 8, 2024

tomvothecoder commented Nov 8, 2024

chengzhuzhang commented Nov 8, 2024 • edited Loading

chengzhuzhang commented Nov 8, 2024 • edited Loading

tomvothecoder commented Nov 8, 2024

Possible issue

chengzhuzhang commented Nov 8, 2024

chengzhuzhang commented Nov 9, 2024 • edited Loading

tomvothecoder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomvothecoder commented Nov 11, 2024

chengzhuzhang commented Nov 11, 2024 • edited Loading

tomvothecoder commented Nov 11, 2024 • edited Loading

chengzhuzhang commented Nov 11, 2024

tomvothecoder commented Jan 15, 2025 • edited Loading

tomvothecoder commented Jan 16, 2025

chengzhuzhang commented Oct 29, 2024 •

edited

Loading

chengzhuzhang commented Nov 5, 2024 •

edited

Loading

chengzhuzhang Dec 13, 2024 •

edited

Loading

chengzhuzhang commented Nov 8, 2024 •

edited

Loading

chengzhuzhang commented Nov 8, 2024 •

edited

Loading

chengzhuzhang commented Nov 9, 2024 •

edited

Loading

chengzhuzhang commented Nov 11, 2024 •

edited

Loading

tomvothecoder commented Nov 11, 2024 •

edited

Loading

tomvothecoder commented Jan 15, 2025 •

edited

Loading