Merge branch 'main' of github.com:Geet-George/halodrops into quicklooks

.github/workflows/python-package.yml

@@ -1,4 +1,4 @@
-# This workflow will install Python dependencies using conda, run tests and lint with a variety of Python versions
+# This workflow will install Python dependencies using mamba, run tests and check pre-commit
 
 name: Python package
 
@@ -14,30 +14,23 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
-      matrix:
-        python-version: ["3.10","3.11"]
 
     steps:
     - uses: actions/checkout@v3
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v3
+    - name: Set Swap Space
+      uses: pierotofy/set-swap-space@master
       with:
-        python-version: ${{ matrix.python-version }}
-    - name: Install dependencies
-      run: |
-        $CONDA/bin/conda env update --file environment.yml --name base
-    - name: Lint with flake8
-      run: |
-        $CONDA/bin/conda install flake8
-        # stop the build if there are Python syntax errors or undefined names
-        $CONDA/bin/flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        $CONDA/bin/flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-    - name: Test with pytest
+        swap-size-gb: 10
+    - uses: mamba-org/setup-micromamba@v1
+      with:
+        micromamba-version: 'latest'
+        init-shell: bash
+        generate-run-shell: true
+        environment-name: testenv
+        environment-file: environment.yml
+    - name: Run pytest in micromamba environment
       run: |
-        $CONDA/bin/conda install conda-build
-        $CONDA/bin/conda develop ./src
-        $CONDA/bin/conda install pytest
-        $CONDA/bin/pytest --import-mode=importlib .
+        micromamba install -n testenv pytest
+        micromamba run -n testenv pytest --import-mode=importlib .
     - name: pre-commit
       uses: pre-commit/action@v3.0.0

docs/source/explanation/for_developers.md

@@ -0,0 +1,52 @@
+# For Developers
+
+## How does `halodrops.__init__` work?
+
+The idea is to minimize the decision-making on the user's part. The user should be able to run the package with minimal configuration. The package should be able to handle the rest. For this, all functions in the package should ideally have all arguments with default values. The user can override these default values by providing non-default values in a configuration file.
+
+However, some arguments cannot have default values (e.g. `data_directory` or `flight_id`). These arguments are mandatory and must be provided by the user within a `MANDATORY` section in the configuration file. This means that functions in the package that have the same mandatory arguments must always use the same argument name across the whole package (e.g. `data_directory` should not be called by a different function as `data_dir`).
+
+The package handles these non-default and mandatory values and executes the functions with the provided arguments.
+
+This `__init__` script thus retrieves mandatory and non-default values from the configuration file for functions within the package and its subpackages, and executes those functions with the retrieved arguments.
+
+### Functions
+`__init__` defines several functions:
+
+`get_all_defaults(package)`: This function retrieves the default values of all functions within a package and its subpackages. It returns a dictionary where the keys are the fully qualified names of the functions, and the values are dictionaries containing the default values of the function parameters.
+
+`nondefault_values_from_config(config, default_dict)`: This function retrieves non-default argument values from a configuration file. It returns a dictionary containing the non-default arguments and their corresponding values based on the config file.
+
+`get_mandatory_args(function)`: This function retrieves a list of all arguments that do not have a default value for a given function.
+
+`get_mandatory_values_from_config(config, mandatory_args)`: This function extracts mandatory values from the 'MANDATORY' section of a configuration file. It returns a dictionary where the keys are the argument names and the values are the corresponding values from the config file.
+
+### Main Execution
+The script's main execution begins by parsing command-line arguments to get the path to a configuration file. It then checks if the provided directory and file exist. If they do, it reads the configuration file.
+
+Next, it retrieves the default values for all functions within the halodrops package using the `get_all_defaults` function. It then retrieves the non-default values from the configuration file using the `nondefault_values_from_config` function.
+
+The script then defines a list of functions to execute. For each function, it retrieves the non-default arguments from the previously retrieved non-default values. If the function has mandatory arguments, it retrieves their values from the configuration file using the `get_mandatory_values_from_config` function.
+
+Finally, the script executes each function with the retrieved arguments.
+
+### Usage
+To use this script, you need to provide a configuration file that contains the non-default and mandatory values for the functions you want to execute. The configuration file should should have a `MANDATORY` section and a separate for each function where non-default values are to be provided, where the section name is the fully qualified name of the function (e.g.`api.qc.run`). Each section should contain options for the function arguments, where the option names are the argument names and the option values are the argument values.
+
+An example config file would look like
+
+```ini
+[MANDATORY]
+data_directory = /path/to/data
+flight_id = 20220401
+
+[api.qc.run]
+arg1 = nondefault1
+arg2 = nondefault2
+```
+
+You can run the script from the command line simply by running `halodrops` or optionally with the `-c` or `--config_file_path` option followed by the path to your configuration file. For example:
+
+```bash
+halodrops -c /path/to/config/file
+```

docs/source/explanation/index.md

@@ -6,4 +6,5 @@
 
 performance_of_sondes
 reconditioning
+for_developers
 ```

environment.yml

@@ -3,7 +3,7 @@ channels:
   - conda-forge
   - defaults
 dependencies:
-  - python=3.10
+  - python>=3.10
   - sphinx
   - ca-certificates
   - openssl
@@ -22,4 +22,3 @@ dependencies:
     - sphinx-book-theme
     - sphinx-last-updated-by-git
     - -e .
-    - git+https://github.com/Geet-George/gogoesgone.git@main

halodrops.cfg

@@ -1,2 +1,7 @@
-[PATHS]
-data-directory = "./Data/"
+[MANDATORY]
+data_directory = /Users/geetgeorge/Documents/Work/Research/Projects/Pre-TUD/AC3/Dropsondes/Data/Data
+flight_id = 20220401
+
+[api.qc.run2]
+arg1 = nondefault1
+arg2 = nondefault2

src/halodrops/__init__.py

@@ -1,4 +1,5 @@
 import logging
+from pathlib import Path
 
 # create halodrops logger
 logger = logging.getLogger("halodrops")
@@ -27,9 +28,13 @@
 logger.addHandler(fh_debug)
 logger.addHandler(ch)
 
+import configparser
+from . import pipeline as pi
+
 
 def main():
     import argparse
+    import halodrops
 
     parser = argparse.ArgumentParser("Arguments")
 
@@ -63,3 +68,5 @@ def main():
 
             config = configparser.ConfigParser()
             config.read(config_file_path)
+
+    pi.run_pipeline(pi.pipeline, config)

src/halodrops/helper/__init__.py

@@ -0,0 +1,153 @@
+import numpy as np
+
+# Keys in l2_variables should be variable names in aspen_ds attribute of Sonde object
+l2_variables = {
+    "u_wind": {
+        "rename_to": "u",
+        "attributes": {
+            "standard_name": "eastward_wind",
+            "long_name": "u component of winds",
+            "units": "m s-1",
+            "coordinates": "time lon lat alt",
+        },
+    },
+    "v_wind": {
+        "rename_to": "v",
+        "attributes": {
+            "standard_name": "northward_wind",
+            "long_name": "v component of winds",
+            "units": "m s-1",
+            "coordinates": "time lon lat alt",
+        },
+    },
+    "tdry": {
+        "rename_to": "ta",
+        "attributes": {
+            "standard_name": "air_temperature",
+            "long_name": "air temperature",
+            "units": "K",
+            "coordinates": "time lon lat alt",
+        },
+    },
+    "pres": {
+        "rename_to": "p",
+        "attributes": {
+            "standard_name": "air_pressure",
+            "long_name": "atmospheric pressure",
+            "units": "Pa",
+            "coordinates": "time lon lat alt",
+        },
+    },
+    "rh": {
+        "attributes": {
+            "standard_name": "relative_humidity",
+            "long_name": "relative humidity",
+            "units": "",
+            "coordinates": "time lon lat alt",
+        }
+    },
+    "lat": {
+        "attributes": {
+            "standard_name": "latitude",
+            "long_name": "latitude",
+            "units": "degree_north",
+            "axis": "Y",
+        }
+    },
+    "lon": {
+        "attributes": {
+            "standard_name": "longitude",
+            "long_name": "longitude",
+            "units": "degree_east",
+            "axis": "X",
+        }
+    },
+    "time": {
+        "attributes": {
+            "standard_name": "time",
+            "long_name": "time of recorded measurement",
+            "axis": "T",
+        }
+    },
+    "gpsalt": {
+        "attributes": {
+            "standard_name": "altitude",
+            "long_name": "gps reported altitude above MSL",
+            "units": "m",
+            "axis": "Z",
+            "positive": "up",
+        }
+    },
+}
+
+encoding_variables = {
+    "time": {"units": "seconds since 1970-01-01", "dtype": "float"},
+}
+
+variable_compression_properties = dict(
+    zlib=True,
+    complevel=4,
+    fletcher32=True,
+    _FillValue=np.finfo("float32").max,
+)
+
+l2_flight_attributes_map = {
+    "True Air Speed (m/s)": "true_air_speed_(ms-1)",
+    "Ground Speed (m/s)": "ground_speed_(ms-1)",
+    "Software Notes": "AVAPS_software_notes",
+    "Format Notes": "AVAPS_format_notes",
+    "True Heading (deg)": "true_heading_(deg)",
+    "Ground Track (deg)": "ground_track_(deg)",
+    "Longitude (deg)": "aircraft_longitude_(deg_E)",
+    "Latitude (deg)": "aircraft_latitude_(deg_N)",
+    "MSL Altitude (m)": "aircraft_msl_altitude_(m)",
+    "Geopotential Altitude (m)": "aircraft_geopotential_altitude_(m)",
+}
+
+
+l2_filename_template = (
+    "HALO-(AC)3_{platform}_{launch_time}_{flight_id}_{serial_id}_Level_2.nc"
+)
+
+
+def get_bool(s):
+    if isinstance(s, bool):
+        return s
+    elif isinstance(s, int):
+        return bool(s)
+    elif isinstance(s, str):
+        lower_s = s.lower()
+        if lower_s == "true":
+            return True
+        elif lower_s == "false":
+            return False
+        elif lower_s in ["0", "1"]:
+            return bool(int(lower_s))
+        else:
+            raise ValueError(f"Cannot convert {s} to boolean")
+    else:
+        raise ValueError(f"Cannot convert {s} to boolean")
+
+
+def convert_rh_to_si(value):
+    """convert RH from % to fraction"""
+    return value / 100
+
+
+def convert_pres_to_si(value):
+    """convert pressure from hPa to Pa"""
+    return value * 100
+
+
+def convert_tdry_to_si(value):
+    """convert temperature from C to K"""
+    return value + 273.15
+
+
+def get_si_converter_function_based_on_var(var_name):
+    """get the function to convert a variable to SI units based on its name"""
+    func_name = f"convert_{var_name}_to_si"
+    func = globals().get(func_name, None)
+    if func is None:
+        raise ValueError(f"No function named {func_name} found in the module")
+    return func