bring file from main branch. (#58)

* bring file from main branch.

* update nomad exmaple tests.

* Rename pytest step Test pynxtools data conversion.

* Nonmd exmample ready adn passing ci/cd.

* CI: install nomad with infrastructure extra and test also with python 3.12

* run nomad test again python 3.12

* depend on pynxtools>0.9

* update pynxtools in dev-requirements

---------

Co-authored-by: Lukas Pielsticker <50139597+lukaspie@users.noreply.github.com>

.github/workflows/pytest.yml

@@ -6,6 +6,9 @@ on:
   pull_request:
     branches: [main]
 
+env:
+  UV_SYSTEM_PYTHON: true
+
 jobs:
   pytest:
     runs-on: ubuntu-latest
@@ -15,20 +18,29 @@ jobs:
         python_version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
           submodules: recursive
       - name: Set up Python ${{matrix.python_version}}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{matrix.python_version}}
-      - name: Install dependencies
+      - name: Install uv
         run: |
           python -m pip install --upgrade pip
-      - name: Install package
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+      - name: Install nomad
+        if: "${{ matrix.python_version != '3.8'}}"
         run: |
-          pip install ".[dev]"
-      - name: Test with pytest
+          uv pip install nomad-lab[infrastructure]@git+https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR.git
+      - name: Install package and dependencies
         run: |
-          pytest tests
+          uv pip install ".[dev]"
+      - name: Test pynxtools data conversion
+        run: |
+          pytest tests/test_reader.py
+      - name: Test Nomad Examples
+        if: "${{ matrix.python_version != '3.8'}}"
+        run: |
+          pytest tests/test_nomad_examples.py

MANIFEST.in

@@ -1 +1,5 @@
-recursive-include pynxtools_stm/examples **
+prune *
+exclude *
+recursive-include pynxtools_stm **
+include README.md pyproject.toml LICENSE CITATION.cff
+graft pynxtools_stm/nomad/examples

dev-requirements.txt

@@ -208,7 +208,7 @@ pymdown-extensions==10.11.2
     #   mkdocs-material
     #   mkdocs-mermaid2-plugin
     #   mkdocstrings
-pynxtools @ git+https://github.com/FAIRmat-NFDI/pynxtools.git@master
+pynxtools==0.9.0
     # via
     #   -r requirements.txt
     #   pynxtools-stm (pyproject.toml)
@@ -260,7 +260,7 @@ ruamel-yaml==0.18.6
     #   pykwalify
 ruamel-yaml-clib==0.2.12
     # via ruamel-yaml
-ruff==0.7.0
+ruff==0.8.0
     # via pynxtools-stm (pyproject.toml)
 scipy==1.12.0
     # via

docs/explanation/reader-explanation.md

@@ -1,7 +1,7 @@
 # STS Reader
 The prime purpose of the reader is to transform data from measurement files into community-defined concepts constructed by the SPM community which allows experimentalists to store, organize, search, analyze, and share experimental data (only within the [NOMAD](https://nomad-lab.eu/nomad-lab/) platform) among the scientific communities. The reader builds on the [NXsts](https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/contributed_definitions/NXsts.nxdl.xml) application definition and needs an experimental file, a config file and a eln file to transform the experimental data into the [NXsts](https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/contributed_definitions/NXsts.nxdl.xml) application concepts. 
 
-## Supproted File Formats and File Versions
+## Supported File Formats and File Versions
 
 - Can parse Scanning Tunneling Spectroscopy (STS) from
     - `.dat` file format from Nanonis: 

docs/reference/reference.md

@@ -7,3 +7,31 @@ The reader is fully integrated into the [NOMAD](https://nomad-lab.eu/prod/v1/gui
 ## NeXus
 The reader is using the [NXsts](https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/NXsts.html#nxsts) ([GitHub page](https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/contributed_definitions/NXsts.nxdl.xml)) application definition (as a standardized schema) which is developed using the [NeXus ontology](https://www.nexusformat.org/) ([GitHub page](https://github.com/FAIRmat-NFDI/nexus_definitions/tree/fairmat)). To understand the application definition, properly understanding NeXus ontology can be helpful.
 
+## STS Reader
+The main goal of the STS Reader is to transform different file formats from diverse STS lab into STS community standard [STS application definition](https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/contributed_definitions/NXsts.nxdl.xml), community defined template that define indivisual concept associated with STS experiment constructed by SPM community.
+## STS Example
+There are diverse examples for several versions (Generic 5e and Generic 4.5) of the Nanonis software for STS experiments at [https://gitlab.mpcdf.mpg.de](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-remote-tools-hub/-/tree/develop/docker/sts). But, to utilize that examples one must have an account at https://gitlab.mpcdf.mpg.de. If still you want to try out the examples from the STS reader, please reach out to [Rubel Mozumder](mozumder@physik.hu-berlin.de) or the docker container (discussed below).
+
+To get a detailed overview of the sts reader implementation visit [pynxtools-stm](https://github.com/FAIRmat-NFDI/pynxtools-stm).
+
+## STS docker image
+STS docker image contains all prerequisite tools (e.g. jupyter-notebook) and library to run STS reader. To use the image user needs to [install docker engine](https://docs.docker.com/engine/install/).
+
+STS Image: `gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-remote-tools-hub/sts-jupyter:latest`
+
+To run the STS image as a docker container copy the code below in a file `docker-compose.yaml`
+
+```docker
+# docker-compose.yaml
+version: "3.9"
+services:
+    sts:
+        image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-remote-tools-hub/sts-jupyter:latest
+        ports:
+            - 8888:8888
+        volumes:
+            - ./example:/home/jovyan/work_dir
+        working_dir: /home/jovyan/work_dir
+```
+
+and launch the file from the same directory with `docker compose up` command.

docs/tutorial/nomad-tutorial.md

@@ -1,4 +1,12 @@
 # Upload Your Data in NOMAD
 
 !!! warning "Attention"
-    This part is on progress.
+    This part is on progress.
+
+## Examples in NOMAD
+Currently, we have two examples in NOMAD, one for STS (Scanning Tunneling Spectroscopy) and one for STM (Scanning Tunneling Microscopy).
+### STS Example
+The [STS example](../../src/pynxtools_mpes/nomad/examples/spm/sts) Scanning Tunneling Spectroscopy (STS) from Scanning Probe Microscopy family. It can transform the data from files generated from nanonis instrument and software into a data format defined by NeXus application definition NXsts.
+
+### STM Example
+The [STM example](../../src/pynxtools_mpes/nomad/examples/spm/stm) Scanning Tunneling Microscopy (STM) from Scanning Probe Microscopy family. It can transform the data from files generated from nanonis instrument and software into a data format defined by NeXus application definition NXstm.

pynxtools_stm/nomad/entrypoints.py

@@ -0,0 +1,35 @@
+#
+# Copyright The NOMAD Authors.
+#
+# This file is part of NOMAD. See https://nomad-lab.eu for further info.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Entry points for STS and STM examples."""
+
+from nomad.config.models.plugins import ExampleUploadEntryPoint
+
+sts_example = ExampleUploadEntryPoint(
+    title="Scanning Tunneling Spectroscopy (STS)",
+    category="SPM experiments examples",
+    description="""This example presents the capabilities of the NOMAD platform to store standardized Scanning Tunneling Spectroscopy (sts) data.""",
+    plugin_package="pynxtools_stm",
+    resources=["nomad/examples/sts/*"],
+)
+stm_example = ExampleUploadEntryPoint(
+    title="Scanning Tunneling Microscopy (STM)",
+    category="SPM experiments examples",
+    description="""This example presents the capabilities of the NOMAD platform to store standardized Scanning Tunneling Microscopy (stm)""",
+    plugin_package="pynxtools_stm",
+    resources=["nomad/examples/stm/*"],
+)

pynxtools_stm/nomad/examples/stm/README.md

@@ -0,0 +1,10 @@
+# STM Reader
+This is an example of STM in NOMAD. The example can be utilised to upload the experimental data from STM experiments (but you still need to modify the config file according to the data structure in experimental file). In order to understand the reader functionality and config file structure, please visit the [reader documentation ](https://fairmat-nfdi.github.io/pynxtools-stm/).
+
+The prime purpose of the reader is to transform data from measurement files into community-defined concepts constructed by the SPM community which allows experimentalists to store, organize, search, analyze, and share experimental data (only within the [NOMAD](https://nomad-lab.eu/nomad-lab/) platform) among the scientific communities. The reader builds on the [NXsts](https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/contributed_definitions/NXsts.nxdl.xml) application definition and needs an experimental file, a config file and a eln file to transform the experimental data into the [NXsts](https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/contributed_definitions/NXsts.nxdl.xml) application concepts.
+
+## Supported File Formats and File Versions
+
+- Can parse Scanning Tunneling Microscopy (STM) from
+    - `.sxm` file format from Nanonis:
+        - Versions: Generic 5e, Generic 4.5