HiTRes Inversion Options

[hanchawn]

Description:

• Add HiTRes options to hbmcmc set up to allow for creating mf timeseries using the HiTRes procedures
• Add edits to name to filter files imported for the date (avoid importing all matching files and then cropping data) and make some processes faster e.g. option to create timeseries in fp_sensitivity so timeseries_HiTRes doesn't need to be run both there and in footprints_data_merge
• Add overwrite and uncertainty options to flux.write

resolves issues #124 & #126

Type of Change:
[ ] Bug fix
[X] New feature
[ ] Code-breaking change

Checklist:
[ ] Code documentation has been updated if needed
[ ] If new tests are needed, they have been added
[X] All tests pass
[X] No merge conflicts
[X] /CHANGELOG.md has been updated if change is significant

[hanchawn]

In terms of the inversion, the HiTRes specific processes are only used in the set up i.e. where it calls footprints_data_merge & fp_sensitivity - although I agree that the test coverage for the HiTRes part of name.py is not great
I thought that the existing hbmcmc tests would cover the rest as it's not HiTRes specific
The current tests pass for the changes I made to name.py, including a test for name.timeseries_HiTRes (but it doesn't test the other HiTRes parts of name.py)

[rt17603]

In terms of the inversion, the HiTRes specific processes are only used in the set up i.e. where it calls footprints_data_merge & fp_sensitivity - although I agree that the test coverage for the HiTRes part of name.py is not great I thought that the existing hbmcmc tests would cover the rest as it's not HiTRes specific The current tests pass for the changes I made to name.py, including a test for name.timeseries_HiTRes (but it doesn't test the other HiTRes parts of name.py)

Ok, but given that this PR is to integrate the HiTRes options into the hbmcmc code it would be useful to have even a very simple example which demonstrates that integration works. Even if we're happy each individual part works it's nice to know that the steps can and do work together.

[hanchawn]

I haven't added this test as I'm not sure how to write a test to run an inversion without it being very inefficient - from what I could see of the current hbmcmc tests they mostly use a toy model which doesn't use fpdm inputs (I could be wrong though).

It would be good to have this merged in so that we could add a version number to the APO paper, I can add an issue that some version of the test should be added later.

There are 4 tests failing but these are also failing in develop

[rt17603]

Based on previous code - is it no longer expected that fp_directory will be a dictionary for 'HiTRes' option?

[hanchawn]

I've never used a dictionary for it as the HiTRes footprints are stored in the same area as the integrated ones now, do you think it would be better to change the code to make that an option again, or just take out reference to is being a dictionary?

[hanchawn]

A lot of the code would need changing to use a dictionary for it, and I don't think you'd often have a reason for importing both so I think it would be best just to change the documentation
(The way the footprints are processed now means that the HiTRes code now only needs the HiTRes ones)

[rt17603]

I think the question is around backwards compatibility - thinking about it I think the dictionary option was because we used to have to use separate integrated and HiTRes files. If it's not too much work if may be good to allow it to be included as an option? Otherwise, maybe you could just add a comment to say this mode is no longer supported.

[rt17603]

Yeah ok I think that's fine - seems like a thing to add to the change log just so we know when the change happened.

[hanchawn]

Sounds good

[rt17603]

I've reviewed this as much as I can given the time constraints and I think this looks really good overall.

You're correct that there are currently failing tests in develop anyway so these aren't being influenced by your changes. I think you're right it's best to get this in but as you say if an issue could be added to say that this needs to be tested that would be very helpful. At the very least were there some examples of how this can be run with example inputs etc.?

As a group we also should really think about getting this integrated into the openghg-inversions repository but that's a future problem and better to get this integrated for now.

[hanchawn]

Thank you Rachel, I appreciate it's a big PR so you'e done a thorough job in the circumstances!

I never got to the point of running a HiTRes inversion so I don't have example runs of that, but my APO models rely on some of the other changes to run.

Eric has added the HiTRes option to the openghg-inversions code, and is going to make any other changes that need to go in for that.