Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix qc handling #136

Merged
merged 4 commits into from
Jan 28, 2025
Merged

fix qc handling #136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
32ff983
add variable qcs to level3
hgloeckner Jan 15, 2025
52d0eb0
allow to add qc flags to vars with different names
hgloeckner Jan 15, 2025
d673edc
move N m calc in pipeline
hgloeckner Jan 17, 2025
0d54277
set keep default to none
hgloeckner Jan 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions pydropsonde/helper/quality.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,19 +437,21 @@ def get_details(self, variable):
)
return self.qc_by_var.get(variable).get("qc_details"), attrs

def add_variable_flags_to_ds(self, ds, variable, details=True):
def add_variable_flags_to_ds(self, ds, variable, add_to=None, details=True):
if add_to is None:
add_to = variable
name = f"{variable}_qc"
value, attrs = self.get_byte_array(variable)
ds = ds.assign({name: value})
ds[name].attrs.update(attrs)
ds = hx.add_ancillary_var(ds, variable, name)
ds = hx.add_ancillary_var(ds, add_to, name)
# get detail
if details:
qc_dict, attrs = self.get_details(variable)
for key in list(qc_dict.keys()):
ds = ds.assign({key: qc_dict.get(key)})
ds[key].attrs.update(attrs.get(key))
ds = hx.add_ancillary_var(ds, variable, key)
ds = hx.add_ancillary_var(ds, add_to, key)
tmieslinger marked this conversation as resolved.
Show resolved Hide resolved

return ds

Expand Down
4 changes: 2 additions & 2 deletions pydropsonde/pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -542,15 +542,15 @@ def run_pipeline(pipeline: dict, config: configparser.ConfigParser):
"remove_non_mono_incr_alt",
"swap_alt_dimension",
"interpolate_alt",
"get_N_m_values",
"add_Nm_to_vars",
"recalc_rh_and_ta",
"add_iwv",
"add_ids",
"add_wind",
"add_attributes_as_var",
"make_attr_coordinates",
"add_qc_to_interim_l3",
"get_N_m_values",
"add_Nm_to_vars",
"add_globals_l3",
"add_expected_coords",
"save_interim_l3",
Expand Down
35 changes: 30 additions & 5 deletions pydropsonde/processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1496,7 +1496,7 @@ def make_attr_coordinates(self):
self.interim_l3_ds = ds.assign_coords(new_coords)
return self

def add_qc_to_interim_l3(self, keep=["sonde_qc"]):
def add_qc_to_interim_l3(self, keep=None):
"""
Add quality control flags to the interim Level 3 dataset.

Expand All @@ -1510,6 +1510,7 @@ def add_qc_to_interim_l3(self, keep=["sonde_qc"]):
Returns:
self: The instance with updated `interim_l3_ds` including quality control flags.
"""
ds = self.interim_l3_ds
if keep is None:
keep = []
elif keep == "all":
Expand All @@ -1518,11 +1519,35 @@ def add_qc_to_interim_l3(self, keep=["sonde_qc"]):
+ list(self.qc.qc_details.keys())
+ ["low_physics", "alt_near_gpsalt"]
)
elif isinstance(keep, str):
keep = keep.split(",")
keep = keep + ["sonde_id"]
else:
for var in ds.variables:
if var != "sonde_id":
ds[var].attrs.pop("ancillary_variables", None)
Comment on lines +1524 to +1525
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot follow here: do you remove the key "ancillary_variables" from the dict?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. if not all qc variables from level2 to level3, then the entry in the attributes of the variables is deleted (otherwise it would be referring to qc variables that are no longer in the dataset).
The attribute is just a string, so I thought it would be easier to completely remove it first and then add it again later with the qc flags that actually made it into the dataset.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense :)

if keep is None:
keep = []
elif keep == "var_flags":
keep = [f"{var}_qc" for var in list(self.qc.qc_by_var.keys())] + [
"sonde_qc"
]
for var in self.qc.qc_by_var.keys():
ds = hx.add_ancillary_var(ds, var, var + "_qc")
if (not np.isin("q", self.qc.qc_vars)) and np.isin(
"rh", self.qc.qc_vars
):
ds = hx.add_ancillary_var(ds, "q", "rh_qc")
if (not np.isin("theta", self.qc.qc_vars)) and np.isin(
"ta", self.qc.qc_vars
):
ds = hx.add_ancillary_var(ds, "theta", "ta_qc")
Comment on lines +1532 to +1541
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I can follow here. However, from the previous changes to add_variable_flags_to_ds I had thought that you'd be able to handle those explicitly stated cases with add_variable_flags_to_ds(add_to=...)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

theoretically yes, but the if is necessary anyways right? And this is already the case where all the qc details are dropped, so it doesn't make a difference. (or am i missing something?).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, maybe in other words: the change above is such that one can add all the qc flags from one variable to another. For example, rh_qc, rh_near_surface_count, rh_profile_fullness can be added with this one liner as ancillaries to q. It also calculates the aggregated qc flag for a variable, which is not necessary here since the aggregated flag already exists.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I understand now better the different QCs :) thanks!


else:
warnings.warn(
"your keep argument for the qc flags in level 3 is not valid, no qc added"
)
keep = []
keep = keep + ["sonde_qc"]
ds_qc = self.interim_l2_ds[keep].expand_dims("sonde_id")
self.interim_l3_ds = xr.merge([self.interim_l3_ds, ds_qc])
self.interim_l3_ds = xr.merge([ds, ds_qc])

return self

Expand Down