Merge pull request #600 from dbekaert/dev

v0.4.5

.github/workflows/build.yml

@@ -12,13 +12,13 @@ on:
 
 jobs:
   call-version-info-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-version-info.yml@v0.8.1
+    uses: ASFHyP3/actions/.github/workflows/reusable-version-info.yml@v0.8.2
     with:
       python_version: '3.10'
 
   call-docker-ghcr-workflow:
     needs: call-version-info-workflow
-    uses: ASFHyP3/actions/.github/workflows/reusable-docker-ghcr.yml@v0.8.1
+    uses: ASFHyP3/actions/.github/workflows/reusable-docker-ghcr.yml@v0.8.2
     with:
       version_tag: ${{ needs.call-version-info-workflow.outputs.version_tag }}
       release_branch: main

.github/workflows/changelog.yml

@@ -13,6 +13,6 @@ on:
 
 jobs:
   call-changelog-check-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-changelog-check.yml@v0.8.1
+    uses: ASFHyP3/actions/.github/workflows/reusable-changelog-check.yml@v0.8.2
     secrets:
       USER_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/deploy-docs.yml

@@ -10,7 +10,7 @@ jobs:
     name: Build site and deploy
     runs-on: "ubuntu-latest"
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
 

.github/workflows/labeled-pr.yml

@@ -12,4 +12,4 @@ on:
 
 jobs:
   call-labeled-pr-check-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-labeled-pr-check.yml@v0.8.1
+    uses: ASFHyP3/actions/.github/workflows/reusable-labeled-pr-check.yml@v0.8.2

.github/workflows/release.yml

@@ -7,7 +7,7 @@ on:
 
 jobs:
   call-release-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-release.yml@v0.8.1
+    uses: ASFHyP3/actions/.github/workflows/reusable-release.yml@v0.8.2
     with:
       release_prefix: RAiDER
       develop_branch: dev

.github/workflows/tag.yml

@@ -7,7 +7,7 @@ on:
 
 jobs:
   call-bump-version-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-bump-version.yml@v0.8.1
+    uses: ASFHyP3/actions/.github/workflows/reusable-bump-version.yml@v0.8.2
     with:
       user: dbekaert
       email: bekaertdavid@gmail.com

CHANGELOG.md

@@ -6,6 +6,39 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.5]
+
+## Fixes
+* [#583](https://github.com/dbekaert/RAiDER/issues/583): it appears that since the issues with geo2rdr cropped up during our processing campaign, there has been a new release of ISCE3 that resolves these failures with `geo2rdr` and the time interpolation that uses this ISCE3 routine.
+* [#584](https://github.com/dbekaert/RAiDER/issues/584): failed Raider step function in hyp3 job submission when HRRR model times are not available (even within the valid model range) - to resolve, we check availability of files when delay workflow called with a) azimuth_grid_interpolation and b) input to workflow is GUNW. If weather model files are unavailable and the GUNW is on s3, do nothing to GUNW (i.e. do not add tropo delay) and exit successfully. If weather model files are unavailable and the GUNW is on local disk, raise `ValueError`
+* [#587](https://github.com/dbekaert/RAiDER/issues/587): similar to 584 except added here to the mix is control flow in RAiDER.py passes over numerous exceptions in workflow. This is fixed identically as above.
+* [#596](https://github.com/dbekaert/RAiDER/issues/596): the "prefix" for aws does not include the final netcdf file name, just the sub-directories in the bucket and therefore extra logic must be added to determine the GUNW netcdf file name (and the assocaited reference/secondary dates). We proceed by downloading the data which is needed regardless. Tests are updated.
+
+## Removed
+* Removes `update` option (either `True` or `False`) from calcGUNW workflow which asks whether the GUNW should be updated or not. In existing code, it was not being used/applied, i.e. previous workflow always updated GUNW. Removed input arguments related from respective functions so that it can be updated later.
+
+## Added
+* Allow for Hyp3 GUNW workflow with HRRR (i.e. specifying a gunw path in s3) to successfully exit if any of the HRRR model times required for `azimuth-time-grid` interpolation (which is default interpolatin method) are not available when using bucket inputs (i.e. only on the cloud)
+* For GUNW workflow, when model is HRRR, azimuth_time_grid interpolation used, and using a local GUNW, if requisite weather model files are not available for  raise a ValueError (before processing)
+* Raise a value error if non-unique dates are given in the function `get_inverse_weights_for_dates` in `s1_azimuth_timing.py` 
+* Added metadata provenance for each delay layer that is included in GUNW and the cube workflow generally in `calcDelays` including:
+   * `model_times_used` - the weather models used and interpolated for the delay calculation
+   * `interpolation_method` - whether `none`, `center_time`, or `azimuth_time_grid` methods were used - see description in `calcDelayGUNW`
+   * `scene_center_time` - the center time in which the associated SAR image was acquired
+* Stages GMAO data for GUNW testing of correct dataset update i.e. in the test `test_GUNW_dataset_update`.
+* Stages HRRR data for `test_HRRR_ztd` test.
+* Ensures ISCE3 is `>=0.15.0`
+* Uses correct HyP3 S3 prefix conventions and filename suffix within test patches to improve readability of what tests are mocking (see comments in #597).
+
+## Changed
+* Get only 2 or 3 model times required for azimuth-time-interpolation (previously obtained all 3 as it was easier to implement) - this ensures slightly less failures associated with HRRR availability. Importantly, if a acquisition time occurs during a model time, then we order by distance to the reference time and how early it occurs (so earlier times come first if two times are equidistant to the aquisition time).
+* Made test names in `test_GUNW.py` more descriptive
+* Numpy docstrings and general linting to modified function including removing variables that were not being accessed
+* hrrr_download to ensure that `hybrid` coordinate is obtained regardless how herbie returns datacubes and ensures test_HRRR_ztd passes consistently
+   * Remove the command line call in `test_HRRR_ztd.py` and call using the python mock up of CLI for better error handling and data mocking.
+* Return xarray.Dataset types for RAiDER.calcGUNW.tropo_gunw_slc and RAiDER.raider.calcDelayGUNW for easier inspection and testing
+* Fixes tests for checking availability of HRRR due Issue #596 (above).
+
 ## [0.4.4]
 
 ## Fixes

environment.yml

@@ -25,7 +25,7 @@ dependencies:
  - herbie-data
  - h5netcdf
  - hyp3lib
- - isce3>=0.9.0
+ - isce3>=0.15.0
  - jsonschema==3.2.0  # this is for ASF DAAC ingest schema validation
  - lxml
  - matplotlib

test/conftest.py

@@ -81,10 +81,8 @@ def weather_model_dict_for_azimuth_time_test():
     test_data = TEST_DIR / 'gunw_azimuth_test_data' / 'weather_files'
     return {'HRRR': [test_data / 'HRRR_2021_07_23_T02_00_00_33N_36N_120W_115W.nc',
                      test_data / 'HRRR_2021_07_23_T01_00_00_33N_36N_120W_115W.nc',
-                     test_data / 'HRRR_2021_07_23_T03_00_00_33N_36N_120W_115W.nc',
                      test_data / 'HRRR_2021_07_11_T02_00_00_33N_36N_120W_115W.nc',
                      test_data / 'HRRR_2021_07_11_T01_00_00_33N_36N_120W_115W.nc',
-                     test_data / 'HRRR_2021_07_11_T03_00_00_33N_36N_120W_115W.nc'
                      ]}
 
 
@@ -107,4 +105,48 @@ def orbit_paths_for_duplicate_orbit_xml_test():
                         'S1A_OPER_AUX_POEORB_OPOD_20230413T080643_V20230323T225942_20230325T005942.EOF',
                         'S1A_OPER_AUX_POEORB_OPOD_20230413T080643_V20230323T225942_20230325T005942.EOF',
                         'S1A_OPER_AUX_POEORB_OPOD_20230412T080821_V20230322T225942_20230324T005942.EOF']
-    return [test_data / fn for fn in orbit_file_names]
+    return [test_data / fn for fn in orbit_file_names]
+
+
+@pytest.fixture(scope='session')
+def weather_model_dict_for_gunw_integration_test():
+    """Order is important here; will be in chronological order with respect to closest date times.
+
+    Generate via:
+    ```
+    from RAiDER.processWM import prepareWeatherModel
+    from RAiDER.models import GMAO
+    import datetime
+
+    model = GMAO()
+    datetimes = [datetime.datetime(2020, 1, 30, 12, 0),
+                 datetime.datetime(2020, 1, 30, 15, 0),
+                 datetime.datetime(2020, 1, 24, 12, 0),
+                 datetime.datetime(2020, 1, 24, 15, 0)]
+    bounds = [32.5, 35.5, -119.8, -115.7]
+    wmfiles = [prepareWeatherModel(model, dt, bounds) for dt in datetimes]
+    ```
+    """
+    test_data = TEST_DIR / 'gunw_test_data' / 'weather_files'
+    return {'GMAO': [test_data / 'GMAO_2020_01_30_T12_00_00_32N_36N_121W_114W.nc',
+                     test_data / 'GMAO_2020_01_30_T15_00_00_32N_36N_121W_114W.nc',
+                     test_data / 'GMAO_2020_01_24_T12_00_00_32N_36N_121W_114W.nc',
+                     test_data / 'GMAO_2020_01_24_T15_00_00_32N_36N_121W_114W.nc']
+           }
+
+@pytest.fixture(scope='session')
+def data_for_hrrr_ztd():
+    '''Obtained via:
+    ```
+    from RAiDER.processWM import prepareWeatherModel
+    from RAiDER.models import HRRR
+    import datetime
+
+    model = HRRR()
+    datetimes = [datetime.datetime(2020, 1, 1, 12)]
+    bounds = [36, 37, -92, -91]
+    wmfiles = [prepareWeatherModel(model, dt, bounds) for dt in datetimes]
+    ```
+    '''
+    test_data_dir = TEST_DIR / 'scenario_1' / 'HRRR_ztd_test'
+    return test_data_dir / 'HRRR_2020_01_01_T12_00_00_35N_38N_93W_90W.nc'