From b3be8813d0e30045d84007a8674369c5f7637c5e Mon Sep 17 00:00:00 2001
From: Mark Nestor Costantini <mnc33@cam.ac.uk>
Date: Thu, 13 Feb 2025 15:24:02 +0000
Subject: [PATCH] added docs for inconsistent closure tests

---
 .../tutorials/inconsistent_closuretests.rst   | 95 +++++++++++++++++++
 doc/sphinx/source/tutorials/index.rst         |  1 +
 2 files changed, 96 insertions(+)
 create mode 100644 doc/sphinx/source/tutorials/inconsistent_closuretests.rst

diff --git a/doc/sphinx/source/tutorials/inconsistent_closuretests.rst b/doc/sphinx/source/tutorials/inconsistent_closuretests.rst
new file mode 100644
index 0000000000..f5538783e8
--- /dev/null
+++ b/doc/sphinx/source/tutorials/inconsistent_closuretests.rst
@@ -0,0 +1,95 @@
+How to run an inconsistent closure test
+=======================================
+
+Modeling Inconsistencies
+------------------------
+In a realistic situation it may happen that some source of experimental systematics is overlooked 
+or underestimated for a given dataset. In this case, the measured experimental values for this dataset 
+may deviate from the true value by an amount that is not reflected by the experimental covariance matrix. 
+This will then generate tensions between this dataset and the rest of the data. 
+We call such a dataset “inconsistent”. It is interesting to ask how the NNPDF methodology behaves in such case,
+and whether the inconsistency can be detected.
+
+In order to study this situation in a closure test, we model the inconsistency as follow. 
+We separate off the uncorrelated and correlated parts of the experimental covariance matrix 
+
+.. math::
+
+    (C)_{ij} = \delta_{ij} \sigma_i^{(\rm uncorr)} \sigma_j^{(\rm uncorr)} + \sum_{k=1}^{N_{\rm corr}} \sigma_{i,k}^{(\rm corr)}\sigma^{(\rm corr)}_{j,k}
+
+where :math:`\sigma_i^{(\rm uncorr)}` and :math:`\sigma_{i,k}^{(\rm corr)}` denote respectively the uncorrelated and correlated systematics
+for data point :math:`i`. 
+We then define a rescaled covariance matrix 
+
+.. math::
+
+    (C^{\lambda})_{ij} = \delta_{ij} \sigma_i^{(\rm uncorr)} \sigma_j^{(\rm uncorr)} + \sum_{k=1}^{N_{\rm corr}} \lambda_{i,k}\sigma_{i,k}^{(\rm corr)} \lambda_{j,k}\sigma^{(\rm corr)}_{j,k},
+
+in which correlated uncertainties have been rescaled.
+In the closure test workflow we then generate Level 1 pseudodata using :math:`C`, namely 
+
+.. math::
+    L_1 = L_0 + \eta
+
+with :math:`\eta \sim \mathcal{N}(0,C)`. The inconsistency is introduced at Level 2 when fitting the PDF
+by using the rescaled covariance matrix :math:`C^{\lambda}` to propagate data uncertainties to the model space:  
+
+.. math::
+    L_2 = L_1 + \epsilon^{\lambda}.
+
+
+
+Running an inconsistent closure test
+------------------------------------
+
+To run a closure test we require a standard closure test runcard. 
+As already explained in the previous section the specification which controls 
+closure test specific behaviour can be found under ``closuretest``.
+An example of a typical level 1 or level 2 ``closuretest`` specification is given by
+
+.. code:: yaml
+
+  closuretest:
+    filterseed  : 0   # Random seed to be used in filtering data partitions
+    fakedata    : true
+    fakepdf     : MMHT2014nnlo68cl
+    fakenoise   : true
+
+In order to run an inconsistent closure test, we need to first specify that the closure 
+test should be inconsistent by adding the key ``inconsistent_fakedata`` to the ``closuretest``
+specification.
+
+.. code:: yaml
+
+  closuretest:
+    filterseed  : 0   # Random seed to be used in filtering data partitions
+    fakedata    : true
+    fakepdf     : MMHT2014nnlo68cl
+    fakenoise   : true
+    inconsistent_fakedata : true
+
+
+In order to specify which datasets should be made inconsistent in the closure test
+we need to add a new specification.
+For instance
+
+.. code:: yaml
+
+    inconsistent_data_settings:
+
+        treatment_names: [MULT]
+        names_uncertainties: [CORR, SPECIAL]
+
+        inconsistent_datasets:
+            - HERA_NC_318GEV_EM-SIGMARED
+            - HERA_NC_251GEV_EP-SIGMARED
+            - HERA_NC_300GEV_EP-SIGMARED
+            - HERA_NC_318GEV_EP-SIGMARED
+
+        sys_rescaling_factor: 0.0
+
+specifies that the inconsistency should be applied to the datasets
+``HERA_NC_318GEV_EM-SIGMARED``, ``HERA_NC_251GEV_EP-SIGMARED``,
+``HERA_NC_300GEV_EP-SIGMARED`` and ``HERA_NC_318GEV_EP-SIGMARED`` by multiplying
+the MULT, CORR and SPECIAL (correlated intra-datasets) uncertainties by a factor of 0.0.
+
diff --git a/doc/sphinx/source/tutorials/index.rst b/doc/sphinx/source/tutorials/index.rst
index c6c20abf63..f8cb48fe2b 100644
--- a/doc/sphinx/source/tutorials/index.rst
+++ b/doc/sphinx/source/tutorials/index.rst
@@ -37,6 +37,7 @@ Closure tests
    :maxdepth: 1
 
    ./closuretest.rst
+   ./inconsistent_closuretests.rst
    ./closureestimators.rst
 
 Special PDF sets