Add profile extent qc

[hgloeckner]

add a test to check that the profile extends (almost) to the aircraft

[tmieslinger]

The implementation of this new QC check seems reasonable. I'd mainly suggest to rename all "_extend" instances to "_extent" as my first intuition was that this function wants to extend the profile instead of checking the extent.
The QC flag profile_extend seems to indeed happen in 12 cases in the current L3_qc data on IPFS. Once even combined with the flag near_surface :)
You told be before, but why exactly do we need this flag?

[hgloeckner]

The implementation of this new QC check seems reasonable. I'd mainly suggest to rename all "__extend" instances to "__extent" as my first intuition was that this function wants to extend the profile instead of checking the extent. The QC flag profile_extend seems to indeed happen in 12 cases in the current L3_qc data on IPFS. Once even combined with the flag near_surface :) You told be before, but why exactly do we need this flag?

I don't really understand what you mean here. the profile_extent is independent of the near_surface qc. The former checks whether the profile reaches up to the aircraft (at least sufficiently so) and the other checks that there at least 50 values close to the surface. in the {var}_qc variables, all qcs for that variable are combined in this byte-version, maybe that's where they appear together?

[tmieslinger]

The implementation of this new QC check seems reasonable. I'd mainly suggest to rename all "__extend" instances to "__extent" as my first intuition was that this function wants to extend the profile instead of checking the extent. The QC flag profile_extend seems to indeed happen in 12 cases in the current L3_qc data on IPFS. Once even combined with the flag near_surface :) You told be before, but why exactly do we need this flag?

I don't really understand what you mean here. the profile_extent is independent of the near_surface qc. The former checks whether the profile reaches up to the aircraft (at least sufficiently so) and the other checks that there at least 50 values close to the surface. in the {var}_qc variables, all qcs for that variable are combined in this byte-version, maybe that's where they appear together?

Sorry, I mixed by statements :) I was wondering when or where we need to know whether the profile extends all the way to the aircraft. Is it for the integrated variables like IWV to be comparable? I am wondering because the aircraft flew at different hight levels anyway. So when is it a problem if the sonde measurements start >10% below the aircraft altitude?

[hgloeckner]

The implementation of this new QC check seems reasonable. I'd mainly suggest to rename all "__extend" instances to "__extent" as my first intuition was that this function wants to extend the profile instead of checking the extent. The QC flag profile_extend seems to indeed happen in 12 cases in the current L3_qc data on IPFS. Once even combined with the flag near_surface :) You told be before, but why exactly do we need this flag?

I don't really understand what you mean here. the profile_extent is independent of the near_surface qc. The former checks whether the profile reaches up to the aircraft (at least sufficiently so) and the other checks that there at least 50 values close to the surface. in the {var}_qc variables, all qcs for that variable are combined in this byte-version, maybe that's where they appear together?

Sorry, I mixed by statements :) I was wondering when or where we need to know whether the profile extends all the way to the aircraft. Is it for the integrated variables like IWV to be comparable? I am wondering because the aircraft flew at different hight levels anyway. So when is it a problem if the sonde measurements start >10% below the aircraft altitude?

Ah okay :) Yes, it is basically only necessary for the IWV calculation. We do have different aircraft heights, but somehow, there should still be some cutoff when it does not make sense to calculate it. I think, this test only targets one sonde that is not targeted otherwise for orcestra, which for some reason only has measurements in the last 600m. Maybe we can come up with a better way to flag sondes like that?

[tmieslinger]

The implementation of this new QC check seems reasonable. I'd mainly suggest to rename all "__extend" instances to "__extent" as my first intuition was that this function wants to extend the profile instead of checking the extent. The QC flag profile_extend seems to indeed happen in 12 cases in the current L3_qc data on IPFS. Once even combined with the flag near_surface :) You told be before, but why exactly do we need this flag?

I don't really understand what you mean here. the profile_extent is independent of the near_surface qc. The former checks whether the profile reaches up to the aircraft (at least sufficiently so) and the other checks that there at least 50 values close to the surface. in the {var}_qc variables, all qcs for that variable are combined in this byte-version, maybe that's where they appear together?

Sorry, I mixed by statements :) I was wondering when or where we need to know whether the profile extends all the way to the aircraft. Is it for the integrated variables like IWV to be comparable? I am wondering because the aircraft flew at different hight levels anyway. So when is it a problem if the sonde measurements start >10% below the aircraft altitude?

Ah okay :) Yes, it is basically only necessary for the IWV calculation. We do have different aircraft heights, but somehow, there should still be some cutoff when it does not make sense to calculate it. I think, this test only targets one sonde that is not targeted otherwise for orcestra, which for some reason only has measurements in the last 600m. Maybe we can come up with a better way to flag sondes like that?

Ok, when plotting IWV dependent on altitude we could define a range in altitude that adds most to the total integral. THe highest contribution should come from the BL with decreasing contributions as you go up. Do you think it could make sense to rather have a test for IWV (in the tropics) that will apply a simple fixed height threshold, e.g. the profile has to have measurements up to 10km?

[tmieslinger]

From personal conversation we agreed to split the two points mentioned above:

1. providing a flag value that hints to the extent of a profile. Maybe the profile_fullness flag should be renamed to profile_data_density or similar to not confuse with profile_extent.
2. checking for integrated variables such as IWV that there are datapoints throughout the profile up to at least 8km (?). But the latter can be specific to the variable in question.

[hgloeckner]

From personal conversation we agreed to split the two points mentioned above:

1. providing a flag value that hints to the extent of a profile. Maybe the `profile_fullness` flag should be renamed to `profile_data_density` or similar to not confuse with `profile_extent`.

2. checking for integrated variables such as IWV that there are datapoints throughout the profile up to at least 8km (?). But the latter can be specific to the variable in question.

I used profile_sparsity since we calculate density later. maybe that could lead to confusion.

Other than that, now the extent is dependent on an absolute height value again. It is set to 8000 as a default, but can be changed in the config.

[tmieslinger]

I tested the changes and the renaming seems to be consistent and enter the Level3 files. Should the dropsonde.cfg be changed accordingly to say

[processor.Sonde.filter_qc_fail]
filter_flags = profile_sparcity,near_surface_coverage,alt_near_gpsalt