Merge pull request #112 from nordic-institute/OPMONDEV-181-minimum-fixes

feat: Ability to disable certificate verification during connecting to CS. 
docs: update README and collector docs.
fix: fix anonymizer failing tests.

Refs: OPMONDEV-181

README.md

@@ -36,12 +36,14 @@ Instructions on setting up the MongoDB can be found in the [Database Module](./d
 
 The modules should be set up in the following order:
 
-1. [Collector](./docs/collector_module.md) (before others)
-2. [Corrector](./docs/corrector_module.md) (after Collector, before others)
-3. [Reports](./docs/reports_module.md) (optional, after previous)
-4. [Opendata](./docs/opendata_module.md) (optional, after previous)
-5. [Networking](./docs/networking_module.md) (optional, after Opendata)
-6. [Opendata Collector module](./docs/opendata_collector_module.md) (optional, after Opendata)
+ - [Database module](./docs/database_module.md)
+ - [Collector module](./docs/collector_module.md)
+ - [Corrector module](./docs/corrector_module.md)
+ - [Reports module](./docs/reports_module.md)
+ - [Anonymizer module](./docs/anonymizer_module.md)
+ - [Opendata module](./docs/opendata_module.md)
+ - [Networking/Visualizer module](./docs/networking_module.md)
+ - [Opendata Collector module](./docs/opendata_collector_module.md)
 
 ## Programming language
 

anonymizer_module/metrics_statistics/statistics_manager.py

@@ -37,9 +37,9 @@ def collect_statistics(settings: dict, logger: Logger, output_only: bool = False
 
     statistics: StatisticalData = {
         **requests_counts,
-        **{'member_count': json.dumps(member_counts)},
-        **{'service_count': len(services)},
-        **{'service_request_count': json.dumps(services_counts)}
+        'member_count': json.dumps(member_counts),
+        'service_count': len(services),
+        'service_request_count': json.dumps(services_counts)
     }
     if output_only:
         logger.info('Metrics statistical data:\n\n%s', pformat(statistics, indent=2, width=2))

collector_module/etc/settings.yaml

@@ -67,6 +67,12 @@ xroad:
     protocol: http://
     host: <FILL>
     timeout: 10
+    # path to client's certificate
+    tls-client-certificate:
+    # path to client's private key
+    tls-client-key:
+    # path to server's certificate, or False to disable server certificate verification
+    tls-server-certificate:
 
   # Security server used to contact
   security-server:
@@ -77,7 +83,7 @@ xroad:
     tls-client-certificate:
     # path to client's private key
     tls-client-key:
-    # path to server's certificate
+    # path to server's certificate, or False to disable server certificate verification
     tls-server-certificate:
 
   # X-Road service configuration used to fetch operational monitoring requests.

collector_module/opmon_collector/central_server_client.py

@@ -28,8 +28,14 @@
 
 class CentralServerClient:
     def __init__(self, xroad_settings, logger_m):
-        self.url = f"{xroad_settings['central-server']['protocol']}{xroad_settings['central-server']['host']}"
-        self.timeout = xroad_settings['central-server']['timeout']
+        central_server_settings = xroad_settings['central-server']
+        self.url = f"{central_server_settings['protocol']}{central_server_settings['host']}"
+        self.timeout = central_server_settings['timeout']
+        self.server_cert = central_server_settings.get('tls-server-certificate')
+        self.client_cert = (
+            central_server_settings.get('tls-client-certificate'),
+            central_server_settings.get('tls-client-key')
+        )
         self.logger_m = logger_m
 
     def get_security_servers(self):
@@ -43,13 +49,15 @@ def get_security_servers(self):
     def _get_shared_params(self):
         internal_conf_url = f'{self.url}/internalconf'
 
-        global_conf = requests.get(internal_conf_url, timeout=self.timeout)
+        global_conf = requests.get(internal_conf_url, timeout=self.timeout, cert=self.client_cert,
+                                   verify=self.server_cert)
         global_conf.raise_for_status()
         #  NB! re.search global configuration regex might be changed
         # according version naming or other future naming conventions
         data = global_conf.content.decode('utf-8')
         s = re.search(r'Content-location: (/V\d+/\d+/shared-params.xml)', data)
-        shared_params = requests.get(f'{self.url}{s.group(1)}', timeout=self.timeout)
+        shared_params = requests.get(f'{self.url}{s.group(1)}', timeout=self.timeout,
+                                     cert=self.client_cert, verify=self.server_cert)
         shared_params.raise_for_status()
         return shared_params
 

docs/collector_module.md

@@ -1,6 +1,6 @@
 
-|  [![X-ROAD](img/xroad-metrics-100.png)](https://x-road.global/) | ![European Union / European Regional Development Fund / Investing in your future](img/eu_rdf_100_en.png "Documents that are tagged with EU/SF logos must keep the logos until 1.11.2022. If it has not stated otherwise in the documentation. If new documentation is created  using EU/SF resources the logos must be tagged appropriately so that the deadline for logos could be found.") |
-| :-------------------------------------------------- | -------------------------: |
+| [![X-ROAD](img/xroad-metrics-100.png)](https://x-road.global/) | ![European Union / European Regional Development Fund / Investing in your future](img/eu_rdf_100_en.png "Documents that are tagged with EU/SF logos must keep the logos until 1.11.2022. If it has not stated otherwise in the documentation. If new documentation is created  using EU/SF resources the logos must be tagged appropriately so that the deadline for logos could be found.") |
+|:---------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
 
 # X-Road Metrics - Collector Module
 
@@ -68,13 +68,13 @@ sudo apt-get install xroad-metrics-collector
 ```
 
 The installation package automatically installs following items:
- * xroad-metrics-collector command to run the collector manually
+ * `xroad-metrics-collector` command to run the collector manually
  * Linux user named _xroad-metrics_ and group _xroad-metrics_
  * settings file _/etc/xroad-metrics/collector/settings.yaml_
  * cronjob in _/etc/cron.d/xroad-metrics-collector-cron_ to run collector automatically every three hours
  * log folders to _/var/log/xroad-metrics/collector/_
 
-Only _xroad-metrics_ user can access the settings files and run xroad-metrics-collector command.
+Only _xroad-metrics_ user can access the settings files and run `xroad-metrics-collector` command.
 
 To use collector you need to fill in your X-Road and MongoDB configuration into the settings file.
 Refer to section [Collector Configuration](#collector-configuration)
@@ -92,23 +92,29 @@ To use collector you need to fill in your X-Road and MongoDB configuration into
 ```bash
 sudo vi /etc/xroad-metrics/collector/settings.yaml
 ```
+> [!TIP]  
+> For a complete list of available settings, please refer to this [settings.yaml](../collector_module/etc/settings.yaml) template file.
 
 Settings that the user must fill in:
 * X-Road instance name
-* Central- and Security Server hosts
+* Central and Security Server hosts
 * X-Road client used to collect the monitoring data
 * username and password for the collector module MongoDB user
 
-To run collector for multiple X-Road instances, a settings profile for each instance can be created. For example to have profiles DEV, TEST and PROD create three copies of `setting.yaml`
-file named `settings_DEV.yaml`, `settings_TEST.yaml` and `settings_PROD.yaml`.
-Then fill the profile specific settings to each file and use the --profile
-flag when running xroad-metrics-collector. For example to run using the TEST profile:
-```
-xroad-metrics-collector --profile TEST collect
-```
-
-`xroad-metrics-collector` command searches the settings file first in current working direcrtory, then in
-_/etc/xroad-metrics/collector/_
+#### Configurations for multiple X-Road instances
+
+To run collector for multiple X-Road instances, a settings profile for each instance can be created.   
+1. To have profiles `DEV`, `TEST` and `PROD`, create three copies of `setting.yaml`
+file named `settings_DEV.yaml`, `settings_TEST.yaml` and `settings_PROD.yaml` respectively.
+2. Fill the profile specific settings to each file.
+3. Use the `--profile` flag when running `xroad-metrics-collector`.   
+   For example, to run using the `TEST` profile:
+   ```shell
+   xroad-metrics-collector --profile TEST collect
+   ```
+> [!IMPORTANT]  
+> `xroad-metrics-collector` command searches the settings file first in current working directory, then in
+`/etc/xroad-metrics/collector/`
 
 ### Using client certificate (mTLS) to connect to security server
 
@@ -128,10 +134,11 @@ security-server:
     tls-client-key: /path/to/client.key # path to client's private key
     tls-server-certificate: /path/to/server.crt # path to server's certificate
 ```
-Notes:
-  Client's certificate has to be sent to security server administrator.
-  Server certificate has to be sent by server's administrator and save in client's location.
-  `tls-server-certificate` can be set to `False` to disable server certificate verification.
+
+> [!Note]
+> - Client's certificate has to be sent to security server administrator.
+> - Server certificate has to be sent by server's administrator and save in client's location.
+> - `tls-server-certificate` can be set to `False` to disable server certificate verification.
 
 ### Manual usage
 
@@ -230,7 +237,7 @@ In case of "activity": "collector_end", the "msg" includes values separated by c
 
 The **collector module** log handler is compatible with the logrotate utility. To configure log rotation for the example setup above, create the file:
 
-```
+```bash
 sudo vi /etc/logrotate.d/xroad-metrics-collector
 ```