very first interim_l3 dataset #121
Conversation
|
... and Issue #120 should be solved here as well - for now I just copied the Sonde header and way to do things |
|
Hi @hgloeckner, thanks for starting this. I think it makes a lot of sense, but maybe with some changes? I'm "typing out loud" my thoughts here. There are two main steps (and a whole bunch of sub-steps, see #90) to go from L2 to L3: (a) Prepare L2 data for concatenation (interpolate, cell methods, etc.) - this still happens to (b) Concatenate & incorporate L3 features (attrs, vars, etc.) - this happens to After (a) is done, this will be stored in an attribute (currently To summarize, there will be a bunch of functions between "add_l2_ds" and "prepare_l2_for_gridded" in the pipeline, and we should make I hope these make sense. Otherwise the PR is exactly in the direction where we want to go! :) |
src/halodrops/processor.py
Outdated
|
|
||
| object.__setattr__(self, "_interim_l3_ds", ds) | ||
|
|
||
| return self | ||
|
|
||
|
|
||
| @dataclass(order=True, frozen=True) |
There was a problem hiding this comment.
| @dataclass(order=True, frozen=True) | |
| @dataclass(order=True) |
Let's skip the frozen already for Gridded! For Sonde, it was more important because sonde_id should be unique.
src/halodrops/processor.py
Outdated
| sondes: dict | ||
| # platforms: dict = field(default_factory=dict) | ||
| _: KW_ONLY | ||
| interpolation_grid: np.ndarray = field( |
There was a problem hiding this comment.
These class attributes of interpolation_grid, interpolation_bins and max_gap_fill will make it fixed values, and unable to be changed by user config. I think it is better to have these values as something that can be imported from a helper function. The concat_sondes can then use these as arguments.
| @@ -967,9 +986,32 @@ def add_q_and_theta_to_l2_ds(self): | |||
| else: | |||
| ds = self.l2_ds | |||
|
|
|||
| ds = calc_q_from_rh(ds) | |||
| ds = calc_theta_from_T(ds) | |||
| ds = hh.calc_q_from_rh(ds) | |||
There was a problem hiding this comment.
This should be a different commit. (that's my bad to not include hh, sorry :P)
There was a problem hiding this comment.
yes true. I subcontiously changed it at some point ^^'
|
Your completely right - Nina, Theresa and I will also talk this afternoon. |
hgloeckner
force-pushed
the
create_interim_l3
branch
from
dccf67a to
c333faf
Compare
src/halodrops/pipeline.py
Outdated
| def sondes_to_gridded(sondes: dict) -> xr.Dataset: | ||
| pass | ||
| def sondes_to_gridded(sondes: dict, config: configparser.ConfigParser): | ||
| print(sondes) |
There was a problem hiding this comment.
| print(sondes) |
|
|
||
| def remove_non_mono_incr_alt(self): | ||
| """ | ||
| This function removes the indices in the |
There was a problem hiding this comment.
| This function removes the indices in the | |
| This function removes the value at the indices which have a non-monotonically increasing value in the |
|
I am not really sure if I accidentally deleted something essential from the interpolation method compared to joanne. One downside in the comparison is that my version needs the |
hgloeckner
force-pushed
the
create_interim_l3
branch
2 times, most recently
from
42002b9 to
b95e72b
Compare
|
It's not working if there are sondes without a launch detect yet. Fixing that currently by removing those from the sonde_dictionary. That means that the sondes that don't detect a launch are removed from the dataset, but I guess for level3 that's ok (they still show in level2). Also, are the sondes where the qc fails currently still in there? then we should remove them as well |
|
I get a basic level3 file out of this. However, I am not sure whether we are (correctly) iterating over the different flight IDs. As far as I understood there should be a level3 file for each flight right? I guess we are not doing that currently |
|
I added the q and theta calculations since I needed them for the plot :) |
hgloeckner
force-pushed
the
create_interim_l3
branch
from
f6cfaec to
3b94fbf
Compare
|
hi @hgloeckner, when trying to plot the sondes in a map, I realized that the lon/lat have dimension (gpsalt), when they should also have dimension (sonde_id). I'll try to fix this |
|
I think also, main is not working anymore right now (or only for one path structure) because the last to commits are actually fixes for the path_structure not so much for this branch. |
I put those changes into another pull request so that we can have a working main before we merge this :) |
hgloeckner
force-pushed
the
create_interim_l3
branch
3 times, most recently
from
33b3621 to
3cdd07e
Compare
hgloeckner
force-pushed
the
create_interim_l3
branch
from
3cdd07e to
2766381
Compare
|
Where does the sonde_id currently come from? I guess it should be something containing the platform, flight date, and sonde number (HALO_20240811_s1). |
|
It's the internal serial id. Geet and Tobi answered this to me in Issue #127 |
|
Are you okay with merging? I think it would be nice to have a basic level 3 in main so that we can start further editing from there @Geet-George |
I am not sure I could completely understand the conversation here. So, is the main branch working now? And what does this basic level 3 include? |
I added a first draft for a gridded class that can be used to start l3 creation.
Right now it does some things where I am not sure if that's useful:
launch_timeis made a dimensionlaunch_timeandtimecausing many missing values (those will go away when we switch dimension to someheightin subsequent steps, but maybe the switch should happen before?)true_heading_(deg), but also things likeauthor) are now variables. We should probably add something that looks at them and if it's the same thing for every sonde makes it an attribute again. I just did not want to lose information.It's not meant to be perfect at this point, but maybe we can use it to discuss whether it makes sense like this at all?