-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactor significant portions of the codebase and refactor some land-…
…atm handlers (#213) 1. Refactor `lib.py` (now deleted) - Closes #190 - Replace `run_parallel()` and `run_serial()` with `__main__.py` -> `E3SMtoCMIP._run_parallel()` and `E3SMtoCMIP._run_serial()` - Updated var name `filepaths` to `vars_to_filepaths` for clarity (its a dict mapping var key to list of string paths) - Update args passed to handler method, remove unnecessary args - Add `try` and `except` statement for submitting `pool` jobs to maintain compatibility with MPAS variable handlers, which use different handler method arguments - Replace `handle_variables()`, `get_dimension_data()` and `load_axis()` with `VarHandler.cmorize()` - Delete `handle_simple()` -- _will be re-implemented from scratch in #130_ 3. Update `handler.py` - Update `VarHandler.cmorize()` - Refactored significantly -- extracted logic into smaller, maintainable private methods - Replaced `data` dictionary storing `xr.DataArrays` with `ds` (`xr.Dataset` object) - Distinguish CMOR usage with "cmor" string in function names and python variables - Add support for hybrid sigma variables: `pfull`, `phalf` - Add `VarHandler._cmor_write()` - Add support for CMORizing fixed-time variables - Separate PR for #217 - If time dimension exists, CMORize the variable with all time and time bound values with a single call to `cmor.write()` instead of looping over each time value index and CMORizing each slice -- **this should improve performance and removes the `tqdm` progress bar.** 4. Update `handlers.yaml` - Add `phalf` and `pfull` entries - Closes #115 - Delete `phalf.py` and `pfull.py` - Add `clcalispo` entry - Closes #218 - Delete `clcalipso.py` 5. Update `_formulas.py` - Update all function argument types and return types from `np.ndarray` to `xr.DataArray` - Add formula functions for `pfull` and `phalf` - Add `convert_units()` function, which handles 1-to-1 unit conversions -- replaces `default.default_handlers.write_data()` 6. Refactor `default.py` (now deleted) - Replaced by `VarHandler.cmorize()` and `_formulas.py.convert_units()` 7. Remove `cdms2` and `cdutil` from dependencies in `dev.yml` and `ci.yml`7 8. Clean up legacy code in `clisccp.py` - Separate PR for #218 9. Add `Makefile` for easy access to commonly used commands (e.g., building and installing package)
- Loading branch information
1 parent
8c818fc
commit 12bb1b3
Showing
33 changed files
with
3,408 additions
and
2,905 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| .PHONY: clean clean-test clean-pyc clean-build docs help | ||
| .DEFAULT_GOAL := help | ||
|
|
||
| define BROWSER_PYSCRIPT | ||
| import os, webbrowser, sys | ||
|
|
||
| from urllib.request import pathname2url | ||
|
|
||
| webbrowser.open("file://" + pathname2url(os.path.abspath(sys.argv[1]))) | ||
| endef | ||
| export BROWSER_PYSCRIPT | ||
|
|
||
| define PRINT_HELP_PYSCRIPT | ||
| import re, sys | ||
|
|
||
| for line in sys.stdin: | ||
| match = re.match(r'^([a-zA-Z_-]+):.*?## (.*)$$', line) | ||
| if match: | ||
| target, help = match.groups() | ||
| print("%-20s %s" % (target, help)) | ||
| endef | ||
| export PRINT_HELP_PYSCRIPT | ||
|
|
||
| BROWSER := python -c "$$BROWSER_PYSCRIPT" | ||
|
|
||
| # To run these commands: make <COMMAND> | ||
| # ================================================== | ||
|
|
||
| help: | ||
| @python -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST) | ||
|
|
||
| # Clean local repository | ||
| # ---------------------- | ||
| clean: clean-build clean-pyc clean-test ## remove all build, test, coverage and Python artifacts | ||
|
|
||
| clean-build: ## remove build artifacts | ||
| rm -fr build/ | ||
| rm -fr conda-build/ | ||
| rm -fr dist/ | ||
| rm -fr .eggs/ | ||
| find . -name '*.egg-info' -exec rm -fr {} + | ||
| find . -name '*.egg' -exec rm -f {} + | ||
|
|
||
| clean-pyc: ## remove Python file artifacts | ||
| find . -name '*.pyc' -exec rm -f {} + | ||
| find . -name '*.pyo' -exec rm -f {} + | ||
| find . -name '*~' -exec rm -f {} + | ||
| find . -name '__pycache__' -exec rm -fr {} + | ||
|
|
||
| clean-test: ## remove test and coverage artifacts | ||
| rm -fr tests_coverage_reports/ | ||
| rm -f .coverage | ||
| rm -fr htmlcov/ | ||
| rm -f coverage.xml | ||
| rm -fr .pytest_cache | ||
| rm -rf .mypy_cache | ||
|
|
||
| clean-logs: | ||
| rm cmor_logs/* | ||
| rm logs/* | ||
|
|
||
|
|
||
| # Quality Assurance | ||
| # ---------------------- | ||
| pre-commit: # run pre-commit quality assurance checks | ||
| pre-commit run --all-files | ||
|
|
||
| lint: ## check style with flake8 | ||
| flake8 e3sm_to_cmip tests | ||
|
|
||
| test: ## run tests quickly with the default Python and produces code coverage report | ||
| pytest | ||
| $(BROWSER) tests_coverage_reports/htmlcov/index.html | ||
|
|
||
| # Documentation | ||
| # ---------------------- | ||
| docs: ## generate Sphinx HTML documentation, including API docs | ||
| rm -rf docs/generated | ||
| cd docs && make html | ||
| $(MAKE) -C docs clean | ||
| $(MAKE) -C docs html | ||
| $(BROWSER) docs/_build/html/index.html | ||
|
|
||
| # Build | ||
| # ---------------------- | ||
| install: clean ## install the package to the active Python's site-packages | ||
| python -m pip install . |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.