Merge pull request #23 from forrestfwilliams/develop

Release v0.3.0

.github/workflows/pytest-golden.yml

@@ -0,0 +1,30 @@
+name: Golden Test
+
+on:
+  workflow_dispatch:
+  # pull_request:
+  #   branches:
+  #     - main
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        python_version: ["3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: mamba-org/setup-micromamba@v1
+        with:
+          environment-file: environment.yml
+          create-args: >-
+            python=${{ matrix.python_version }}
+
+      - name: Pytest in conda environment
+        shell: bash -l {0}
+        run: |
+          python -m pip install --no-deps .
+          pytest -m golden

.github/workflows/pytest-integration.yml

@@ -0,0 +1,29 @@
+name: Integration Test
+
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        python_version: ["3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: mamba-org/setup-micromamba@v1
+        with:
+          environment-file: environment.yml
+          create-args: >-
+            python=${{ matrix.python_version }}
+
+      - name: Pytest in conda environment
+        shell: bash -l {0}
+        run: |
+          python -m pip install --no-deps .
+          pytest -m integration

.github/workflows/pytest.yml

@@ -7,10 +7,24 @@ on:
       - develop
 
 jobs:
-  call-pytest-workflow:
-    # Docs: https://github.com/ASFHyP3/actions
-    uses: ASFHyP3/actions/.github/workflows/reusable-pytest.yml@v0.11.0
-    with:
-      local_package_name: burst2safe
-      python_versions: >-
-        ["3.10", "3.11", "3.12"]
+  pytest:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        python_version: ["3.10", "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: mamba-org/setup-micromamba@v1
+        with:
+          environment-file: environment.yml
+          create-args: >-
+            python=${{ matrix.python_version }}
+
+      - name: Pytest in conda environment
+        shell: bash -l {0}
+        run: |
+          python -m pip install --no-deps .
+          pytest --cov=burst2safe

CHANGELOG.md

@@ -6,6 +6,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.0]
+
+### Added
+* Support for IPF >=3.40 RFI annotation files.
+* Support for IPF <=2.90.
+* IPF-specific support files.
+* Calculation of `platformHeading` and `burst/byteOffset` fields.
+
+### Fixed
+* Path information for annotation/measurement files are now are updated when the SAFE path is.
+* Bug when burst widths are different by one pixel
+
+### Changed
+* Test suite to use test data from 2024 (IPF version 3.71).
+
 ## [0.2.0]
 
 ### Added
@@ -33,4 +48,3 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ### Added
 * Initial version of project.
-

README.md

@@ -60,25 +60,25 @@ A given data/metadata component must be recalculated using a process unique to e
 
 For `Merge` components, we have made the best effort to follow the merging instructions outlined in the product specification. While we hope to eventually correctly reconstruct all merged fields, there are some components for whom the implementation is unclear. In these cases, we have set the values of these fields to NULL (`''`) so that downstream processors raise errors instead of using incorrect values. **If any fields we have omitted in this way cause your application to fail, let us know so that we can prioritize its development!**
 
-### Omissions and Nulls
-In some cases, we have not created certain datasets or metadata components because the creation process is unknown to us or is irrelevant for assembled products. This includes datasets such as all datasets in the SAFE `preview` directory the SAFE report PDF included with each SAFE file.
+### Deviations from Specification
+In some cases, we were not able to recreate certain datasets or metadata fields in the exact way that the IPF computes them, because the creation process is unknown to us, or utilizes data that we do not have access to. This includes datasets such as all datasets in the SAFE `preview` directory the SAFE report PDF included with each SAFE file, some metadata fields in the annotation datasets.
 
-All full accounting of omitted datasets and fields can be found below:
+A full accounting of omitted datasets and differing fields can be found below:
 
 * Annotation
     * Noise
-        * No intentional omissions or nulls.
+        * No intentional omissions or deviations.
     * Calibration
-        * No intentional omissions or nulls.
+        * No intentional omissions or deviations
     * RFI
-        * RFI annotation XMLs are currently omitted, but we plan to add this soon.
+        * No intentional omissions or deviations.
     * Product
-        * `generalAnnotation/productInformation/platformHeading` set to `''`.
+        * `generalAnnotation/productInformation/platformHeading`
+            - calculated as average of input Level 1 SLCs, not recalculated from Level-0 slices.
         * `imageAnnotation/imageInformation/azimuthPixelSpacing`
-            - calculated as average of Level 1 SLCs, not slices.
+            - calculated as average of input Level 1 SLCs, not Level-0 slices.
         * `imageAnnotation/imageInformation/imageStatistics/outputDataMean` / `outputDataStdDev`
             - calculated using `np.mean`/`np.std` on valid data.
-        * `swathTiming/burstList/burst/byteOffset` components are set to `''`.
 * Measurement GeoTIFFs
     * Invalid data as denoted by `swathTiming/burstList/burst/firstValidSample` and `lastValidSample` are set to zero. This done by the ASF extractor, not this tool.
     * TIFF tags **that are not GeoTIFF tags** are omitted. See Product Specification Table 3-8 for full list.
@@ -91,8 +91,8 @@ All full accounting of omitted datasets and fields can be found below:
 * SAFE report
     * The SAFE report PDF is omitted.
 
-### Other Known Compatibility Issues
-This tool may not work for SAFEs created with Sentinel-1 IPF version <= v3.19. In the product annotation files, the `height` sub-field of `GeolocationGridPoint` components was not available prior to IPF v3.19. This field is currently used to concatenate geocontrol points for the measurment GeoTIFFs, and thus the tool may fail if these sub-fields are not present.
+### IPF Version Compatibility
+At this time, we are not aware of any compatibility issues with older Sentinel-1 Instrument Processing Facility (IPF) versions. However, if you do encounter any incompatibilities [please open an issue](https://github.com/forrestfwilliams/burst2safe/issues/new), so we can fix it!
 
 ## License
 `burst2safe` is licensed under the BSD 2-Clause License. See the LICENSE file for more details.
@@ -104,7 +104,7 @@ Contributions this project are welcome! If you would like to contribute, please
 Want to talk about `burst2safe`? We would love to hear from you!
 
 Found a bug? Want to request a feature?
-[open an issue](https://github.com/ASFHyP3/asf_tools/issues/new)
+[open an issue](https://github.com/forrestfwilliams/burst2safe/issues/new)
 
 General questions? Suggestions? Or just want to talk to the team?
 [chat with us on burst2safe's discussion page](https://github.com/forrestfwilliams/burst2safe/discussions)

environment.yml

@@ -8,6 +8,7 @@ dependencies:
   - gdal
   - numpy
   - lxml
+  - tifffile
   - asf_search
   # For packaging, and testing
   - pytest

pyproject.toml

@@ -33,6 +33,7 @@ dependencies = [
   "gdal",
   "numpy",
   "lxml",
+  "tifffile",
   "asf_search",
 ]
 
@@ -50,8 +51,8 @@ develop = [
 
 [tool.pytest.ini_options]
 minversion = "6.0"
-addopts = '-ra -q -m "not integration"'
-markers = ["integration"]
+addopts = '-ra -q -m "not integration and not golden"'
+markers = ["integration", "golden"]
 testpaths = ["tests"]
 
 [tool.black]

src/burst2safe/base.py

@@ -29,6 +29,9 @@ def __init__(
 
         self.name = self.inputs[0].tag
         elements = flatten([element.findall('*') for element in self.inputs])
+        if len(elements) == 0:
+            raise ValueError(f'No Sub-elements contained within {self.name}.')
+
         names = drop_duplicates([x.tag for x in elements])
         if len(names) != 1:
             raise ValueError('Elements must contain only one type of subelement.')
@@ -40,6 +43,8 @@ def __init__(
             self.time_field = 'azimuthTime'
         elif 'time' in [x.tag for x in elements[0].iter()]:
             self.time_field = 'time'
+        elif 'noiseSensingTime' in [x.tag for x in elements[0].iter()]:
+            self.time_field = 'noiseSensingTime'
         else:
             raise ValueError('Time field not found in elements.')
 
@@ -236,11 +241,19 @@ def create_data_object(
 
 
 class Annotation:
-    def __init__(self, burst_infos: Iterable[BurstInfo], metadata_type: str, image_number: int):
-        """Initialize the Annotation object."""
+    def __init__(self, burst_infos: Iterable[BurstInfo], metadata_type: str, ipf_version: str, image_number: int):
+        """Initialize the Annotation object.
+
+        Args:
+            burst_infos: The list of burst information objects.
+            metadata_type: The type of metadata to create.
+            ipf_version: The IPF version of the annotation (i.e. 3.71).
+            image_number: The image number of the annotation.
+        """
         self.burst_infos = burst_infos
         self.metadata_type = metadata_type
         self.image_number = image_number
+        self.major_version, self.minor_version = [int(v) for v in ipf_version.split('.')]
         self.metadata_paths = drop_duplicates([x.metadata_path for x in burst_infos])
         self.swath, self.pol = burst_infos[0].swath, burst_infos[0].polarization
         self.start_line = burst_infos[0].burst_index * burst_infos[0].length

src/burst2safe/burst2safe.py

@@ -50,7 +50,7 @@ def find_group(orbit: int, footprint: Polygon, polarizations: Iterable[str]) ->
         if not single_pol:
             # TODO: add Vertex link to error message?
             raise ValueError(
-                f'No results found for orbit {orbit}, footprint {footprint}, and polarization {pol}.'
+                f'No results found for orbit {orbit}, footprint {footprint}, and polarization {pol}. '
                 'Bursts may not populated yet for this group. Check Vertex to confirm.'
             )
         results.extend(list(single_pol))
@@ -153,7 +153,7 @@ def main() -> None:
     parser = ArgumentParser(description=DESCRIPTION)
     parser.add_argument('granules', nargs='*', help='A list of bursts to convert to SAFE')
     parser.add_argument('--orbit', type=int, help='The absolute orbit number of the bursts')
-    parser.add_argument('--bbox', type=float, nargs=4, help='Bounding box of the bursts (S W N E in lat/lon)')
+    parser.add_argument('--bbox', type=float, nargs=4, help='Bounding box of the bursts (W S E N in lat/lon)')
     parser.add_argument('--pols', type=str, nargs='+', help='The polarizations of the bursts (i.e., VV VH)')
     parser.add_argument('--keep-files', action='store_true', default=False, help='Keep the intermediate files')
     args = parser.parse_args()

src/burst2safe/calibration.py

@@ -10,14 +10,15 @@
 class Calibration(Annotation):
     """Class representing a calibration XML."""
 
-    def __init__(self, burst_infos: Iterable[BurstInfo], image_number: int):
+    def __init__(self, burst_infos: Iterable[BurstInfo], ipf_version: str, image_number: int):
         """Create a calibration object.
 
         Args:
             burst_infos: List of BurstInfo objects.
+            ipf_version: The IPF version of the annotation (i.e. 3.71).
             image_number: Image number.
         """
-        super().__init__(burst_infos, 'calibration', image_number)
+        super().__init__(burst_infos, 'calibration', ipf_version, image_number)
         self.calibration_information = None
         self.calibrattion_vector_list = None
 

src/burst2safe/data/support_236/s1-level-1-noise.xsd

@@ -0,0 +1,71 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- $Id: s1-level-1-noise.xsd 9013 2012-12-12 23:41:57Z cthain@DS.MDA.CA $ -->
+<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="unqualified" attributeFormDefault="unqualified">
+   <xsd:include schemaLocation="s1-object-types.xsd"/>
+   <xsd:complexType name="noiseVectorType">
+      <xsd:annotation>
+         <xsd:documentation>Annotation record for noise vectors.</xsd:documentation>
+      </xsd:annotation>
+      <xsd:sequence>
+         <xsd:element name="azimuthTime" type="timeType">
+            <xsd:annotation>
+               <xsd:documentation>Zero Doppler azimuth time at which noise vector applies [UTC].</xsd:documentation>
+            </xsd:annotation>
+         </xsd:element>
+         <xsd:element name="line" type="uint32">
+            <xsd:annotation>
+               <xsd:documentation>Image line at which the calibration vector applies.</xsd:documentation>
+            </xsd:annotation>
+         </xsd:element>
+         <xsd:element name="pixel" type="intArray">
+            <xsd:annotation>
+               <xsd:documentation>Image pixel at which the calibration vector applies. This array contains the count attribute number of integer values (i.e. one value per point in the noise vectors), separated by spaces. The maximum length of this array will be one value for every pixel in an image line, however in general the vectors will be subsampled.</xsd:documentation>
+            </xsd:annotation>
+         </xsd:element>
+         <xsd:element name="noiseLut" type="floatArray">
+            <xsd:annotation>
+               <xsd:documentation>Thermal noise correction vector power values. This array contains the count attribute number of floating point values separated by spaces. </xsd:documentation>
+            </xsd:annotation>
+         </xsd:element>
+      </xsd:sequence>
+   </xsd:complexType>
+   <xsd:complexType name="noiseVectorListType">
+      <xsd:annotation>
+         <xsd:documentation>List of noise vector annotation records.</xsd:documentation>
+      </xsd:annotation>
+      <xsd:sequence>
+         <xsd:element name="noiseVector" type="noiseVectorType" minOccurs="0" maxOccurs="1500">
+            <xsd:annotation>
+               <xsd:documentation>Noise vector. This record contains the thermal noise estimation annotations which can be used to remove or reinstate thermal noise from the image.</xsd:documentation>
+            </xsd:annotation>
+         </xsd:element>
+      </xsd:sequence>
+      <xsd:attribute name="count" type="unsignedInt" use="required">
+         <xsd:annotation>
+            <xsd:documentation>Number of noiseVector records within the list.</xsd:documentation>
+         </xsd:annotation>
+      </xsd:attribute>
+   </xsd:complexType>
+   <xsd:complexType name="l1NoiseVectorType">
+      <xsd:annotation>
+         <xsd:documentation>Annotation record for Sentinel-1 level 1 noise product annotations.</xsd:documentation>
+      </xsd:annotation>
+      <xsd:sequence>
+         <xsd:element name="adsHeader" type="adsHeaderType">
+            <xsd:annotation>
+               <xsd:documentation>ADS header data set record. This DSR contains information that applies to the entire data set.</xsd:documentation>
+            </xsd:annotation>
+         </xsd:element>
+         <xsd:element name="noiseVectorList" type="noiseVectorListType">
+            <xsd:annotation>
+               <xsd:documentation>Noise vector list. This element is a list of noiseVector records that contain the thermal noise estimation for the image MDS. The list contains an entry for each update made along azimuth.</xsd:documentation>
+            </xsd:annotation>
+         </xsd:element>
+      </xsd:sequence>
+   </xsd:complexType>
+   <xsd:element name="noise" type="l1NoiseVectorType">
+      <xsd:annotation>
+         <xsd:documentation>Sentinel-1 level 1 thermal noise level product annotations.</xsd:documentation>
+      </xsd:annotation>
+   </xsd:element>
+</xsd:schema>