New ht thcovmat #2126

achiefa · 2024-07-15T10:11:47Z

This PR allows for the inclusion of theory uncertainties due to the effect of power corrections. The theory covariance matrix is constructed by computing the shifts for the theoretical predictions as done for the MHOUs. The shift is computed at the level of the structure functions. Then, the shifts for the structure functions are combined to reconstruct the shift for the xsec. For this reason, the calculation of the shift depends on the dataset. Currently, only 1-JET and DIS (NC and CC) are supported.

At the level of the runcard, power corrections are specified as follows

theorycovmatconfig:
  point_prescriptions: ["power corrections"]
  pc_parameters:
  - {ht: H2p, yshift: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.0], nodes: [0.0, 0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 1]}
  - {ht: H2d, yshift: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.0], nodes: [0.0, 0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 1]}
  - {ht: HLp, yshift: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.0], nodes: [0.0, 0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 1]}
  - {ht: HLd, yshift: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.0], nodes: [0.0, 0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 1]}
  - {ht: H3p, yshift: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.0], nodes: [0.0, 0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 1]}
  - {ht: H3d, yshift: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.0], nodes: [0.0, 0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 1]}
  - {ht: Hj, yshift: [2.0, 2.0, 2.0, 2.0, 2.0, 2.0], nodes: [0.25, 0.75, 1.25, 1.75, 2.25, 2.75]}
  func_type: "linear" 
  pc_included_procs: ["JETS", "DIS NC", "DIS CC"]
  pc_excluded_exps: [HERA_NC_318GEV_EAVG_CHARM-SIGMARED,
                     HERA_NC_318GEV_EAVG_BOTTOM-SIGMARED]
  pdf: 240701-02-rs-nnpdf40-baseline
  use_thcovmat_in_fitting: true
  use_thcovmat_in_sampling: true

For each process/sf implemented, we need to specify a series of information:

The name of the power correction, under the key ht. This is useful to identify the type of power correction, in particular when computing the posterior.
The values in yshift are the magnitudes of the prior.
nodes contains the points where the prior is shifted

The array pc_included_procs specifies the processes for which the shifts are computed. I've also implemented the possibility to exclude some particular datasets within the selected processes, and this can be done by specifying the names in pc_excluded_exps.

The key func_type is temporary and will be deleted once we decide which function to use to construct the prior.

All the relevant details for this PR are contained in the module higher_twist_functions.py. For each observable for which the shift has to be computed, I implemented a factory that constructs a function which will then compute the shift. I thought it was kind of necessary to make the shift dependent on the shift parameters (yshift and nodes) and on the prescription according to which we vary the parameters (which for now is fixed), and not on the kinematics (I think this is known as currying in computer science).

TO DO

Add documentation
Remove func_type
What happens if power_corrections is specified but the parameters are not given?
Change the name ht to name (?)
Something else?

github-actions · 2024-07-15T11:33:42Z

Greetings from your nice fit 🤖 !
I have good news for you, I just finished my tasks:

Fit Name: NNBOT-2ad60a7b9-2024-07-15
Fit Report wrt master: https://vp.nnpdf.science/f0TNPeX3SlO5uwbJkr7I8Q==
Fit Report wrt latest stable reference: https://vp.nnpdf.science/X6jx-wHfT5iJuz14nPatnQ==
Fit Data: https://data.nnpdf.science/fits/NNBOT-2ad60a7b9-2024-07-15.tar.gz

Check the report carefully, and please buy me a ☕ , or better, a GPU 😉!

RoyStegeman · 2024-10-04T14:17:36Z

n3fit/src/n3fit/layers/DIS.py

+                 fktable_arr,
+                 dataset_name,
+                 boundary_condition=None,
+                 operation_name="NULL",


keyword arguments (kwargs) you don't use at this level, you don't have to specify. That's what **kwargs is for.

RoyStegeman · 2024-10-04T14:25:10Z

n3fit/src/n3fit/layers/DIS.py

+        This function is very similar to `compute_ht_parametrisation` in
+        validphys.theorycovariance.construction.py. However, the latter
+        accounts for shifts in the 5pt prescription. As of now, this function
+        is meant to work only for DIS NC data, using the ABMP16 result.


Maybe reference Eq. 6 of https://arxiv.org/pdf/1701.05838

RoyStegeman · 2024-10-04T14:30:21Z

n3fit/src/n3fit/layers/DIS.py

+        x = self.exp_kinematics['kin1']
+        y = self.exp_kinematics['kin3']
+        Q2 = self.exp_kinematics['kin2']
+        N2, NL = 1#compute_normalisation_by_experiment(self.dataname, x, y, Q2)


Where is this function? Also I think you can do N2=NL=1, but not NL,N2=1

The DIS.py is not in sync with the new way shifts are computed. Hence, that function that adds the shift in the theory predictions can be removed. I'll work on that later on, as we discussed. I should have deleted them.

RoyStegeman · 2024-10-04T14:36:07Z

n3fit/src/n3fit/scripts/vp_setupfit.py

+        if (sam_t0 := file_content.get('sampling')) is not None:
+            SETUPFIT_FIXED_CONFIG['separate_multiplicative'] = sam_t0.get(
+                'separate_multiplicative', False
+            )


Why do we need this here?

I don't remember why I implemented that...I think I can remove it

RoyStegeman · 2024-10-04T14:37:44Z

n3fit/src/n3fit/scripts/n3fit_exec.py

+            # NOTE: from the perspective of the fit scalevar and ht uncertainties are the same since
+            # they are saved under the same name


Ah yes, we may want to start thinking of a long term solution as well. A tree of if-statments that grows exponentially for each source of theory uncertainty (scales, alphas, HT) is not super nice

RoyStegeman · 2024-10-04T14:38:52Z

validphys2/src/validphys/results.py

+# NOTE: Same a `groups_data_by_process` in `construction.py`
 procs_data = collect("data", ("group_dataset_inputs_by_process",))


feel free to delete one (in master)

validphys2/src/validphys/theorycovariance/construction.py

achiefa · 2025-01-14T14:14:04Z

Hi @roy, I've updated the code so that now the covmat for power corrections can be constructed as in the case of scale variations. Please, look and let me know if something is unclear. I'd like if you could double-check the functions that compute the shifts, in particular when normalisation factors and conversion factors are used.

There are few things that I still don't like here and there, but you're free to propose modifications.

achiefa · 2025-01-14T14:16:31Z

validphys2/src/validphys/theorycovariance/construction.py

@@ -47,11 +49,11 @@ def theory_covmat_dataset(results, results_central_bytheoryids, point_prescripti
    return thcovmat


-ProcessInfo = namedtuple("ProcessInfo", ("preds", "namelist", "sizes"))
+ProcessInfo = namedtuple("ProcessInfo", ("preds", "namelist", "sizes", "data", "data_spec"))


I included another key in the namedtuple containing the information that I need to compute the shifts for the power corrections. I'm not sure this is the best solution though.

achiefa · 2025-01-14T14:17:27Z

validphys2/src/validphys/theorycovariance/construction.py

    return s


 @check_correct_theory_combination
-def covs_pt_prescrip(combine_by_type, point_prescription):
+def covs_pt_prescrip(


I had to add additional dependencies here for the power correction case. I don't know if there is a clever solution, maybe extracting these quantities from the context?

achiefa · 2025-01-14T14:48:06Z

validphys2/src/validphys/theorycovariance/construction.py

+    point_prescription,
+    pdf: PDF,
+    power_corr_dict,
+    pc_included_prosc,


I think the tests fail because there is not default for pc_included_procs (typo) and pc_excluded_exps. What do you suggest to do? Should I add two functions in config.py that default these two values when not specified in the runcard?

github-actions · 2025-01-14T15:29:47Z

Greetings from your nice fit 🤖 !
I have good news for you, I just finished my tasks:

Fit Name: NNBOT-701fd3dd4-2025-01-14
Fit Report wrt master: https://vp.nnpdf.science/LhL3COHERwuMwHEEeGvymg==
Fit Report wrt latest stable reference: https://vp.nnpdf.science/-y7-UpCPSFGPxzP_41VXLA==
Fit Data: https://data.nnpdf.science/fits/NNBOT-701fd3dd4-2025-01-14.tar.gz

Check the report carefully, and please buy me a ☕ , or better, a GPU 😉!

achiefa added the run-fit-bot Starts fit bot from a PR. label Jul 15, 2024

achiefa requested a review from RoyStegeman July 15, 2024 10:11

achiefa self-assigned this Jul 15, 2024

scarlehoff marked this pull request as draft July 17, 2024 14:26

achiefa force-pushed the new_ht_thcovmat branch from 58ffdd3 to 89aa862 Compare July 29, 2024 10:09

RoyStegeman reviewed Oct 4, 2024

View reviewed changes

achiefa force-pushed the new_ht_thcovmat branch 2 times, most recently from 28662e8 to 6b71fae Compare October 11, 2024 22:16

RoyStegeman force-pushed the new_ht_thcovmat branch from d75f1b9 to 8e1dc74 Compare November 14, 2024 16:20

achiefa force-pushed the new_ht_thcovmat branch 2 times, most recently from 348be41 to 9ac473c Compare January 14, 2025 14:06

achiefa added run-fit-bot Starts fit bot from a PR. and removed run-fit-bot Starts fit bot from a PR. labels Jan 14, 2025

achiefa commented Jan 14, 2025

View reviewed changes

achiefa removed the run-fit-bot Starts fit bot from a PR. label Jan 14, 2025

achiefa force-pushed the new_ht_thcovmat branch from 9ac473c to 84642cb Compare January 14, 2025 14:18

achiefa commented Jan 14, 2025

View reviewed changes

achiefa force-pushed the new_ht_thcovmat branch from 39f5640 to ef59af1 Compare January 30, 2025 13:14

achiefa marked this pull request as ready for review January 30, 2025 14:13

RoyStegeman force-pushed the new_ht_thcovmat branch from fdab4c9 to 9f286c6 Compare February 14, 2025 14:35

achiefa added 6 commits February 21, 2025 14:42

Copied fro branch 'HT_thcovmat'

faf56ef

Removed version

d89ea0f

Saving progress - not ready

861cfe0

Implemented d/p ratio

4c3700f

Parsing 'separate_multiplicative' in vp_setupfit

48685d0

Minor adjustments

904bc68

achiefa and others added 28 commits February 21, 2025 14:42

Added HT for F2C data (EMC) - removed deprecated function

9e97f08

Corrected EMC data iron target

cfbdd0e

Removed deprecated code

82a7738

Refactoring + DIS CC

1d16adf

Corrected bug - ready for cc test

6d1ee16

Corrected bug - ready

c012798

Removing unnecessary code

819360a

Corrected bug after rebase

78ea3b0

Add normalisation in CC x-secs

1b6dacc

Correct normalisation

95f3c19

Restore n3fit files from master

1b9ac74

remove _PB suffix from process type

ce7ec30

format a bit

7b3cf67

Update for new thcovmat construction + refactor + docstrings

caf3339

Correct typo + example runcard

75708bc

Correct bug

3588e8a

Correct nuclear factors for nuclear targets

7354b87

First implementation of jet data

73e7706

Change pc jet dependence from pT to eta

418e002

Allowing step-function for the prior

773a344

Vectorize step_function + docstring

c299973

Correct bug in step function

5c65682

Correct bug in step function

2541f1f

Produce covs_pt_prescrip

9fdc5c0

Allow different funcs for posterior

19ba3a8

Adjusting linear triangular function

d77bda7

Correct bug in linear function

a127cb9

Implement multiplicative PC for jet

de6497e

achiefa force-pushed the new_ht_thcovmat branch from 3c8b923 to de6497e Compare February 21, 2025 14:47

achiefa requested a review from scarlehoff February 21, 2025 15:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New ht thcovmat #2126

New ht thcovmat #2126

achiefa commented Jul 15, 2024 •

edited

Loading

github-actions bot commented Jul 15, 2024

RoyStegeman Oct 4, 2024

RoyStegeman Oct 4, 2024

RoyStegeman Oct 4, 2024

achiefa Oct 4, 2024

RoyStegeman Oct 4, 2024

achiefa Oct 4, 2024

RoyStegeman Oct 4, 2024

RoyStegeman Oct 4, 2024

achiefa commented Jan 14, 2025

achiefa Jan 14, 2025

achiefa Jan 14, 2025

achiefa Jan 14, 2025

github-actions bot commented Jan 14, 2025

		# NOTE: from the perspective of the fit scalevar and ht uncertainties are the same since
		# they are saved under the same name

		# NOTE: Same a `groups_data_by_process` in `construction.py`
		procs_data = collect("data", ("group_dataset_inputs_by_process",))

New ht thcovmat #2126

Are you sure you want to change the base?

New ht thcovmat #2126

Conversation

achiefa commented Jul 15, 2024 • edited Loading

TO DO

github-actions bot commented Jul 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

achiefa commented Jan 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 14, 2025

achiefa commented Jul 15, 2024 •

edited

Loading