Native EAMxx support #424

golaz · 2023-05-10T23:39:05Z

Initial set of changes for ts and climo tasks to accept native monthly EAMxx files as input. Requires new
unreleased version of ncclimo.

Update: the ncclimo version to support EAMxx has been released and the command is updated in this commit.

chengzhuzhang · 2024-02-07T00:23:39Z

Update some test results for this branch. Below is a minimal .cfg to make e3sm_diags work on Chrysalis with a pre-built e3sm_diags environment that support EAMxx variable names.

[default]
input = /lcrc/group/e3sm/ac.golaz/E3SM/EAMxx/20230919.F2010-SCREAMv1.ne30pg2_ne30pg2.chrysalis
output = /lcrc/group/e3sm/ac.zhang40/E3SM/EAMxx/20230919.F2010-SCREAMv1.ne30pg2_ne30pg2.chrysalis_debug
case = 20230919.F2010-SCREAMv1.ne30pg2_ne30pg2.chrysalis
www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.zhang40/E3SM/EAMxx/tmp
partition = compute
environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
dry_run = False
debug = True

[climo]
active = True
walltime = "1:00:00"
years = "0002:0006:5",

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "run"
  input_component = "eamxx"
  input_files = "scream.monthly.AVERAGE.nmonths_x1"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  frequency = "monthly"

[ts]
active = True
walltime = "0:30:00"
years = "0002:0006:5",

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "run"
  input_component = "eamxx"
  input_files = "scream.monthly.AVERAGE.nmonths_x1"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  frequency = "monthly"
  vars = "" 
#  vars = "precip_liq_surf_mass_flux,precip_ice_surf_mass_flux,surf_evap,surf_sens_flux," 
#SW_flux_dn_at_tom,SW_flux_up_at_tom,LW_flux_up_at_tom,SW_flux_dn_at_bot,SW_flux_up_at_bot,LW_flux_dn@_at_ot,LW_flux_up_at_bot,U,V"

[e3sm_diags]
active = True
grid = '180x360_aave'
reference_data_path = "/lcrc/group/e3sm/diagnostics/observations/Atm/climatology"
sets = "lat_lon","zonal_mean_xy","zonal_mean_2d","polar","meridional_mean_2d","annual_cycle_zonal_mean","zonal_mean_2d_stratosphere", 
short_name = 'EAMxx_test'
output_format = "png",
environment_commands = "source /home/ac.zhang40/y/etc/profile.d/conda.sh; conda activate e3sm_diags_eamxx"
years = "0002:0006:5"

  [[ atm_monthly_180x360_aave ]]
  walltime = "2:00:00"

Initial set of changes for ts and climo tasks to accept native monthly EAMxx files as input. Supported by new version of ncclimo.

chengzhuzhang · 2024-02-08T00:06:46Z

@forsyth2 this PR is ready for review. Though it has some unit tests failure. Could you help integrate this PR?

forsyth2 · 2024-02-14T17:21:04Z

@chengzhuzhang I have the unit tests passing. I'm currently running the integration tests on Chrysalis. Once I get those done, I'll push my commit and then merge this PR.

forsyth2 · 2024-02-14T23:30:53Z

I added a commit for the unit test fixes. I ran zppy for the complete run test, but encountered an error in ts that prevented ilamb from running.

$ cd /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test-424/v2.LR.historical_0201/post/scripts
$ grep -v "OK" *status
ilamb_1850-1851.status:WAITING 470993
ilamb_1852-1853.status:WAITING 470994
ts_land_monthly_1850-1851-0002.status:ERROR (5)
ts_land_monthly_1852-1853-0002.status:ERROR (5)
$ tail -n 30 ts_land_monthly_1850-1851-0002.o470976 
2024-02-14 17:22:42,501 [INFO]: handler.py(cmorize:209) >> lai: Starting CMORizing
2024-02-14 17:22:42,501 [INFO]: handler.py(cmorize:209) >> lai: Starting CMORizing
2024-02-14 17:22:42,501_501:INFO:cmorize:lai: Starting CMORizing
  0%|          | 0/1 [00:00<?, ?it/s]2024-02-14 17:22:42,733 [INFO]: handler.py(_setup_cmor_module:333) >> lai: CMOR setup complete
2024-02-14 17:22:42,733 [INFO]: handler.py(_setup_cmor_module:333) >> lai: CMOR setup complete
2024-02-14 17:22:42,733_733:INFO:_setup_cmor_module:lai: CMOR setup complete
2024-02-14 17:22:42,734 [INFO]: handler.py(cmorize:239) >> lai: loading E3SM variables dict_keys(['LAISHA', 'LAISUN'])
2024-02-14 17:22:42,734 [INFO]: handler.py(cmorize:239) >> lai: loading E3SM variables dict_keys(['LAISHA', 'LAISUN'])
2024-02-14 17:22:42,734_734:INFO:cmorize:lai: loading E3SM variables dict_keys(['LAISHA', 'LAISUN'])
2024-02-14 17:22:42,922 [INFO]: handler.py(cmorize:247) >> lai: creating CMOR variable with CMOR axis objects.
2024-02-14 17:22:42,922 [INFO]: handler.py(cmorize:247) >> lai: creating CMOR variable with CMOR axis objects.
2024-02-14 17:22:42,922_922:INFO:cmorize:lai: creating CMOR variable with CMOR axis objects.
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
100%|██████████| 1/1 [00:00<00:00,  1.88it/s]
2024-02-14 17:22:43,072 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-14 17:22:43,072 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-14 17:22:43,072_072:INFO:_run_parallel:0 of 1 handlers complete
2024-02-14 17:22:43,072 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-14 17:22:43,072 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-14 17:22:43,072_072:ERROR:_run_parallel:lai failed to complete
2024-02-14 17:22:43,072 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-14 17:22:43,072 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-14 17:22:43,072_072:ERROR:_run_parallel:0 of 1 handlers complete
'LAISHA'
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test-424/v2.LR.historical_0201/post/lnd/180x360_aave/cmip_ts/monthly/tmp_ts_land_monthly_1850-1851-0002/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory

chengzhuzhang · 2024-02-14T23:36:17Z

hmm, i don't think this is caused by this PR, and in fact, it looks like the same issue documented here: #543

forsyth2 · 2024-02-15T00:05:35Z

Yes, I noticed that too. I need to do some debugging.

forsyth2 · 2024-02-15T00:05:45Z

Accidentally closed

forsyth2 · 2024-02-15T17:32:57Z

Ah, it looks like a similar cannot stat error was reported in #523. It looks like the variables need to be changed? Or maybe the mapping file?

forsyth2 · 2024-02-15T18:05:27Z

It looks like removing LAISHA,LAISUN does fix the error.

forsyth2 · 2024-02-15T20:01:18Z

Tests are passing, merging.

golaz marked this pull request as draft May 10, 2023 23:40

chengzhuzhang mentioned this pull request Feb 6, 2024

Support monthly average and variable name convention for EAMxx E3SM-Project/e3sm_diags#712

Merged

golaz added 2 commits February 7, 2024 17:29

Native EAMxx support

244d3b1

Initial set of changes for ts and climo tasks to accept native monthly EAMxx files as input. Supported by new version of ncclimo.

Revert use of custom ncclimo executable.

ae2c2c5

chengzhuzhang force-pushed the golaz/eamxx branch from ae6d4e6 to ae2c2c5 Compare February 7, 2024 23:34

fix pre-committing error

f726ec8

chengzhuzhang marked this pull request as ready for review February 8, 2024 00:05

chengzhuzhang changed the title ~~[WIP] Native EAMxx support~~ Native EAMxx support Feb 8, 2024

forsyth2 mentioned this pull request Feb 13, 2024

[Bug]: e3sm_to_cmip exception running bundles on Unified 1.9.2 #543

Closed

Testing fixes

f331654

forsyth2 closed this Feb 15, 2024

forsyth2 reopened this Feb 15, 2024

forsyth2 added 2 commits February 15, 2024 13:55

Integration test fixes

1662b2d

Rerun utils

2b01d21

forsyth2 merged commit a902549 into main Feb 15, 2024
4 checks passed

forsyth2 deleted the golaz/eamxx branch February 15, 2024 20:01

This was referenced Feb 16, 2024

Add lat_lon_land from e3sm_diags for land diagnostics #548

Merged

Address "flaky" ts subtask in complete_run test #549

Closed

forsyth2 mentioned this pull request Jun 17, 2024

Fix clm2 support #602

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native EAMxx support #424

Native EAMxx support #424

golaz commented May 10, 2023 •

edited by chengzhuzhang

Loading

chengzhuzhang commented Feb 7, 2024 •

edited

Loading

chengzhuzhang commented Feb 8, 2024

forsyth2 commented Feb 14, 2024

forsyth2 commented Feb 14, 2024

chengzhuzhang commented Feb 14, 2024 •

edited

Loading

forsyth2 commented Feb 15, 2024

forsyth2 commented Feb 15, 2024

forsyth2 commented Feb 15, 2024

forsyth2 commented Feb 15, 2024

forsyth2 commented Feb 15, 2024

Native EAMxx support #424

Native EAMxx support #424

Conversation

golaz commented May 10, 2023 • edited by chengzhuzhang Loading

chengzhuzhang commented Feb 7, 2024 • edited Loading

chengzhuzhang commented Feb 8, 2024

forsyth2 commented Feb 14, 2024

forsyth2 commented Feb 14, 2024

chengzhuzhang commented Feb 14, 2024 • edited Loading

forsyth2 commented Feb 15, 2024

forsyth2 commented Feb 15, 2024

forsyth2 commented Feb 15, 2024

forsyth2 commented Feb 15, 2024

forsyth2 commented Feb 15, 2024

golaz commented May 10, 2023 •

edited by chengzhuzhang

Loading

chengzhuzhang commented Feb 7, 2024 •

edited

Loading

chengzhuzhang commented Feb 14, 2024 •

edited

Loading