Refactored main and restructured code #31

ray-chew · 2024-12-18T17:24:54Z

__main__.py is now cleaned, and all the initialisation and analysis steps have been moved to their respective modules or subpackages.

…he code

…tructure there are still a lot of broken imports and references, so the code does not run yet.

…_main__.py still untested as I have to move the analysis steps to the DA subpackage before this will run

also created a placeholder for the parallel integrator tests planned

the restructuring continues

DA and blending are not covered in the tests, will look into them after refactoring the flow solver

since I removed all initial conditions except this

…on time

this is with the new ensemble data structure; the flow solver runs; yet to be tested.

…rrays at the end of the flow solver's looping this is because the tests were failing with a discrepancy.

validation step will be moved to dataclass structure

tbnc · 2025-01-06T08:39:58Z

I will review.

Are the docs checks supposed to succeed?

ray-chew · 2025-01-06T10:22:40Z

I will review.

Are the docs checks supposed to succeed?

Thank you. The checks broke due to the renaming of the subpackages. This issue has been resolved with the latest commits. But the pytests are broken now, as I am in the midst of restructuring the diagnostics module to accept more general definitions of test cases.

more cleaning up is due for the diagnostics modules, but at least now the tests are no longer hardcoded. the blending has potential problems with failing tests, will look into this next.

tbnc · 2025-01-09T15:55:15Z

README.md

-However, the codebase is structured such that the user can easily assemble a run script to define their own experiments. Refer to the documentation for the [available APIs](https://ray-chew.github.io/pyBELLA/apis.html).
+To run a simulation:
+```console
+pybella -ic rb -N 1


This does not appear to work, at least out of the box:

[nhad@ac6-101 pyBELLA]$ pybella -ic rb -N 1 Traceback (most recent call last): File "/home/nhad/.local/bin/pybella", line 8, in <module> sys.exit(main()) ^^^^^^ File "/etc/ecmwf/nfs/dh1_home_b/nhad/pyBELLA/src/__main__.py", line 30, in main sim_state = prepare.initialise() ^^^^^^^^^^^^^^^^^^^^ File "/etc/ecmwf/nfs/dh1_home_b/nhad/pyBELLA/src/utils/prepare.py", line 43, in initialise ud.coriolis_strength = np.array(ud.coriolis_strength) ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'UserDataInit' object has no attribute 'coriolis_strength'

tbnc · 2025-01-09T15:58:28Z

run_scripts/test_dycore.py

@@ -1,16 +0,0 @@
-import pytest


Line 74 in the README

pyBELLA/README.md

Line 74 in 2165bed

pytest ./run_scripts/test_dycore.py -v

should take into account the fact that the file has been deleted.

tbnc · 2025-01-09T15:59:04Z

README.md

-However, the codebase is structured such that the user can easily assemble a run script to define their own experiments. Refer to the documentation for the [available APIs](https://ray-chew.github.io/pyBELLA/apis.html).
+To run a simulation:
+```console
+pybella -ic rb -N 1


This does not appear to work, at least not out of the box:

[nhad@ac6-101 pyBELLA]$ pybella -ic rb -N 1 Traceback (most recent call last): File "/home/nhad/.local/bin/pybella", line 8, in <module> sys.exit(main()) ^^^^^^ File "/etc/ecmwf/nfs/dh1_home_b/nhad/pyBELLA/src/__main__.py", line 30, in main sim_state = prepare.initialise() ^^^^^^^^^^^^^^^^^^^^ File "/etc/ecmwf/nfs/dh1_home_b/nhad/pyBELLA/src/utils/prepare.py", line 43, in initialise ud.coriolis_strength = np.array(ud.coriolis_strength) ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'UserDataInit' object has no attribute 'coriolis_strength'

tbnc · 2025-01-09T16:02:09Z

src/data_assimilation/analysis.py

@@ -0,0 +1,123 @@
+import logging


With a view to the forthcoming published version, I would propose removing commented code lines sooner rather than later unless still part of prototyping (which probably should not end up in develop anyway?) or there is another very compelling reason to keep them in.

tbnc · 2025-01-09T16:05:58Z

src/flow_solver/discretisation/time_update.py

-    mpv,
-    t,
+    sst,
+    mem,


Nice and compact

tbnc · 2025-01-10T09:13:40Z

pyproject.toml

-    "PyYAML==6.0",
-    "scipy==1.7.3",
-    "termcolor==2.4.0"
+    "dask",


tbnc · 2025-01-10T09:13:52Z

pyproject.toml

@@ -3,15 +3,15 @@ name = "pyBELLA"
 version = "0.50.1"

 dependencies = [
-    "dask==2022.7.0",
-    "dill==0.3.6",


tbnc · 2025-01-10T09:14:09Z

pyproject.toml

@@ -3,15 +3,15 @@ name = "pyBELLA"
 version = "0.50.1"

 dependencies = [
-    "dask==2022.7.0",
-    "dill==0.3.6",
-    "h5py==3.7.0",


tbnc · 2025-01-10T09:14:29Z

pyproject.toml

-    "dask==2022.7.0",
-    "dill==0.3.6",
-    "h5py==3.7.0",
-    "matplotlib==3.5.1",


tbnc · 2025-01-10T09:14:40Z

pyproject.toml

-    "dill==0.3.6",
-    "h5py==3.7.0",
-    "matplotlib==3.5.1",
-    "numba==0.56.4",


tbnc · 2025-01-10T09:14:53Z

pyproject.toml

-    "h5py==3.7.0",
-    "matplotlib==3.5.1",
-    "numba==0.56.4",
-    "numpy==1.22.1",


tbnc · 2025-01-10T09:15:14Z

pyproject.toml

-    "matplotlib==3.5.1",
-    "numba==0.56.4",
-    "numpy==1.22.1",
-    "PyYAML==6.0",


tbnc · 2025-01-10T09:15:30Z

pyproject.toml

-    "numba==0.56.4",
-    "numpy==1.22.1",
-    "PyYAML==6.0",
-    "scipy==1.7.3",


tbnc · 2025-01-10T09:15:42Z

pyproject.toml

-    "numpy==1.22.1",
-    "PyYAML==6.0",
-    "scipy==1.7.3",
-    "termcolor==2.4.0"


… container

I was not updating mem attributes in the time update while loop. This means that everytime the blending scheme sees mem, it resets the solution fields to the beginning of the assimilation window.

I have not yet checked if the initial blending results are correct; will do this via the diagnostics module, but it does not work yet, as I have not yet generalised the target generation features.

…compare nothing really works yet, because there seems to be a discrepancy between the target saved and the test values computed and compared against the target values. This has something to do with how the time steps are counted. Will need to carry on debugging.

the generalised target generation in the diagnostics module now works, the problem was with writing the output before the mem attributes were correctly updated.

…ted with warm bubble test values blending is still switched off, but this commit allows for blending to be tested as a next step. woohoo.

ray-chew added 10 commits October 3, 2024 11:53

freezing this version as the base comparison for the refactoring of t…

a977ae7

…he code

checkpoint in restructuring and making the package more pythonic in s…

f4ac3b4

…tructure there are still a lot of broken imports and references, so the code does not run yet.

done with restructuring the initialisation and preparation steps in _…

5e995b8

…_main__.py still untested as I have to move the analysis steps to the DA subpackage before this will run

moved blending from main() into its own submodule

91008f4

also created a placeholder for the parallel integrator tests planned

code runs until analysis step

c72f9a9

the restructuring continues

main has been restructured and all three dynamics tests passed

61e48c2

DA and blending are not covered in the tests, will look into them after refactoring the flow solver

updated readme on how to run a single simulation

3141e13

updated changelog

8f5ac66

updated run.py to run rising bubble test case

37cfa63

since I removed all initial conditions except this

Merge branch 'develop' into refactor

91e0b53

ray-chew added the refactor and optimisation label Dec 18, 2024

ray-chew added this to the Refactor milestone Dec 18, 2024

ray-chew requested a review from tbnc December 18, 2024 17:24

ray-chew added 14 commits December 19, 2024 01:37

refactored diagnostics.py a little

527fcfe

cleaned up blending related code in the flow solver

55a8a16

renamed the dycore subpackage to flow_solver for consistency

acdb0bc

updated data structures for a proper sol_ens and simulation integrati…

7f0b676

…on time

updated thermodynamics data structure.

ee51566

done refactoring the preparation and initialisation steps

91a1018

this is with the new ensemble data structure; the flow solver runs; yet to be tested.

updated analysis with new ensemble structure

468f10c

reinstated a few changes in time_update; updating mem with solution a…

23a7817

…rrays at the end of the flow solver's looping this is because the tests were failing with a discrepancy.

updated docs from dycore to flow_solver

5ca4174

updated changelog

bd2ede2

#6 diagnostics now compute either the sum or the norm as a target

a153faf

reinstated run.py with refactored package structure

9b32880

renamed test_dycore to test_flow_solver

7aba2e1

commented out validation attempt in user_data

e6e043d

validation step will be moved to dataclass structure

working towards a first test case for the blending module

2165bed

flow solver tests now work with the new diagnostics data structure

9433f76

more cleaning up is due for the diagnostics modules, but at least now the tests are no longer hardcoded. the blending has potential problems with failing tests, will look into this next.

tbnc requested changes Jan 9, 2025

View reviewed changes

ray-chew changed the title ~~Refactored main andrestructured code~~ Refactored main and restructured code Jan 10, 2025

tbnc requested changes Jan 10, 2025

View reviewed changes

ray-chew added 7 commits January 10, 2025 19:01

cleaned up blending module; do_blending_bef_timestep accepts mem data…

c7ff474

… container

fixed failing flow solver tests

a4c7b5c

I was not updating mem attributes in the time update while loop. This means that everytime the blending scheme sees mem, it resets the solution fields to the beginning of the assimilation window.

blending routine runs with new data structures

8068cd2

I have not yet checked if the initial blending results are correct; will do this via the diagnostics module, but it does not work yet, as I have not yet generalised the target generation features.

minor refactoring of utils.io and utils.prepare

bc67edc

fixed bug with discrepancy between generated target and test values

035e8a0

the generalised target generation in the diagnostics module now works, the problem was with writing the output before the mem attributes were correctly updated.

update all test ICs to support updating target; test_targets.yml upda…

823c1a7

…ted with warm bubble test values blending is still switched off, but this commit allows for blending to be tested as a next step. woohoo.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactored main and restructured code #31

Refactored main and restructured code #31

ray-chew commented Dec 18, 2024

tbnc commented Jan 6, 2025

ray-chew commented Jan 6, 2025

tbnc Jan 9, 2025

tbnc Jan 9, 2025

tbnc Jan 9, 2025

tbnc Jan 9, 2025

tbnc Jan 9, 2025

tbnc Jan 10, 2025

tbnc Jan 10, 2025

tbnc Jan 10, 2025

tbnc Jan 10, 2025

tbnc Jan 10, 2025

tbnc Jan 10, 2025

tbnc Jan 10, 2025

tbnc Jan 10, 2025

tbnc Jan 10, 2025

Refactored main and restructured code #31

Are you sure you want to change the base?

Refactored main and restructured code #31

Conversation

ray-chew commented Dec 18, 2024

tbnc commented Jan 6, 2025

ray-chew commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment