fix qc handling

[hgloeckner]

• fix the ancillary vars in l3
• add the 'all' flag to the example config

[tmieslinger]

Could you help me understand the default behaviour of the function add_qc_to_interim_l3?

1. If nothing is explicitly specified keepwould be keep=["sonde_qc"].
2. you'd end up in the else-block (line1512) and something will happen to all var != "sonde_id" the ds[var].attrs
3. and now, would you not end up again in the else-block line 1533 and get the warning? keep would be reset to keep=[]
4. keep=[] would be appended again by ["sonde_qc"] before ds_qc is created

This seems strange and I am pretty sure I have a bug in my thinking somewhere. Could you go through the default behaviour once and point me to my bug? 😅

[tmieslinger]

I cannot follow here: do you remove the key "ancillary_variables" from the dict?

[hgloeckner]

Yes. if not all qc variables from level2 to level3, then the entry in the attributes of the variables is deleted (otherwise it would be referring to qc variables that are no longer in the dataset).
The attribute is just a string, so I thought it would be easier to completely remove it first and then add it again later with the qc flags that actually made it into the dataset.

[tmieslinger]

Makes sense :)

[tmieslinger]

I think I can follow here. However, from the previous changes to add_variable_flags_to_ds I had thought that you'd be able to handle those explicitly stated cases with add_variable_flags_to_ds(add_to=...)?

[hgloeckner]

theoretically yes, but the if is necessary anyways right? And this is already the case where all the qc details are dropped, so it doesn't make a difference. (or am i missing something?).

[hgloeckner]

sorry, maybe in other words: the change above is such that one can add all the qc flags from one variable to another. For example, rh_qc, rh_near_surface_count, rh_profile_fullness can be added with this one liner as ancillaries to q. It also calculates the aggregated qc flag for a variable, which is not necessary here since the aggregated flag already exists.

[tmieslinger]

I think I understand now better the different QCs :) thanks!

[hgloeckner]

Could you help me understand the default behaviour of the function add_qc_to_interim_l3?

1. If nothing is explicitly specified `keep`would be `keep=["sonde_qc"]`.

2. you'd end up in the `else`-block (line1512) and something will happen to all var != "sonde_id" the `ds[var].attrs`

3. and now, would you not end up again in the `else`-block line 1533 and get the warning? keep would be reset to `keep=[]`

4. `keep=[]` would be appended again by ["sonde_qc"] before ds_qc is created

This seems strange and I am pretty sure I have a bug in my thinking somewhere. Could you go through the default behaviour once and point me to my bug? 😅

You are completly right. the default should be None.

[tmieslinger]

The changes seem fine to me :)
Working my way through the QC also reminded me of the advantage of adding more info on QC handling to the documentation "soonish", maybe when the QC related PRs are merged.

[tmieslinger]

Makes sense :)

[tmieslinger]

I think I understand now better the different QCs :) thanks!