Skip to content

Latest commit

 

History

History
282 lines (200 loc) · 11.7 KB

collector_module.md

File metadata and controls

282 lines (200 loc) · 11.7 KB
X-ROAD European Union / European Regional Development Fund / Investing in your future

X-Road Metrics - Collector Module

License

This document is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/

About

The Collector module is part of X-Road Metrics, which includes the following modules:

The Collector module is responsible to retrieve data from X-Road Security Servers and insert into the database module. The execution of the collector module is performed automatically via a cron job task.

Overall system, its users and rights, processes and directories are designed in a way, that all modules can reside in one server, but also in separate servers. Opmon modules are controlled by unix user 'opmon' in group 'opmon'.

Overall system is also designed in a way, that allows to monitor data from different X-Road instances (e.g. in Estonia there are three instances: ee-dev, ee-test and EE.)

Overall system is also designed in a way, that can be used by X-Road Centre for all X-Road members as well as for Member own monitoring (includes possibilities to monitor also members data exchange partners).

Networking

Outgoing

  • The collector module needs HTTP-access to the X-Road Central Server to get from global configuration list of members Security Servers.
  • The collector module needs HTTP-access to an X-Road Security Server that acts as an Operational Monitoring Client to get the data is collected.
  • The collector module needs access to the Database Module (see ==> Database_Module <==).

Incoming

No incoming connection is needed in the collector module.

Installation

This sections describes the necessary steps to install the collector module on a Ubuntu 20.04 or Ubuntu 22.04 Linux host. For a complete overview of different modules and machines, please refer to the ==> System Architecture <== documentation.

Add X-Road Extensions Package Repository for Ubuntu

wget -qO - https://artifactory.niis.org/api/gpg/key/public | sudo apt-key add -
sudo add-apt-repository 'https://artifactory.niis.org/xroad-extensions-release-deb main'

The following information can be used to verify the key:

  • key hash: 935CC5E7FA5397B171749F80D6E3973B
  • key fingerprint: A01B FE41 B9D8 EAF4 872F A3F1 FB0D 532C 10F6 EC5B
  • 3rd party key server: Ubuntu key server

Install Collector Package

To install xroad-metrics-collector and all dependencies execute the commands below:

sudo apt-get update
sudo apt-get install xroad-metrics-collector

The installation package automatically installs following items:

  • xroad-metrics-collector command to run the collector manually
  • Linux user named xroad-metrics and group xroad-metrics
  • settings file /etc/xroad-metrics/collector/settings.yaml
  • cronjob in /etc/cron.d/xroad-metrics-collector-cron to run collector automatically every three hours
  • log folders to /var/log/xroad-metrics/collector/

Only xroad-metrics user can access the settings files and run xroad-metrics-collector command.

To use collector you need to fill in your X-Road and MongoDB configuration into the settings file. Refer to section Collector Configuration

Usage

Collector Configuration

Before using the Collector module, make sure you have installed and configured the Database_Module and created the MongoDB credentials.

To use collector you need to fill in your X-Road and MongoDB configuration into the settings file. (here, vi is used):

sudo vi /etc/xroad-metrics/collector/settings.yaml

Tip

For a complete list of available settings, please refer to this settings.yaml template file.

Settings that the user must fill in:

  • X-Road instance name
  • Central and Security Server hosts
  • X-Road client used to collect the monitoring data
  • username and password for the collector module MongoDB user

Settings Profiles

To run collector for multiple X-Road instances, a settings profile for each instance can be created.

  1. To have profiles DEV, TEST, and PROD, create three copies of setting.yaml file named settings_DEV.yaml, settings_TEST.yaml and settings_PROD.yaml respectively.
  2. Fill the profile specific settings to each file.
  3. Use the --profile flag when running xroad-metrics-collector.
    For example, to run using the TEST profile: 
    xroad-metrics-collector --profile TEST collect

Important

xroad-metrics-collector command searches the settings file first in current working directory, then in /etc/xroad-metrics/collector/

Using client certificate (mTLS) to connect to security server

Mutual TLS (mTLS) allows a client and a server to identify and authenticate each other by using X.509 certificates. TLS cryptographic protocol provides secure and integral communication between parties.

To use mTLS in collector you need to fill security-server section in your X-Road settings.yaml file.

Example of configuration:

security-server:
    protocol: https://
    host: example.host
    timeout: 60.0
    tls-client-certificate: /path/to/client.crt # path to client's certificate
    tls-client-key: /path/to/client.key # path to client's private key
    tls-server-certificate: /path/to/server.crt # path to server's certificate

Note

  • Client's certificate has to be sent to security server administrator.
  • Server certificate has to be sent by server's administrator and save in client's location.
  • tls-server-certificate can be set to False to disable server certificate verification.

Manual usage

All collector module actions can be executed by calling the xroad-metrics-collector command. Command should be executed as user xroad-metrics so change to that user:

sudo su xroad-metrics

Some example commands:

xroad-metrics-collector list                       # Print available Security Servers to stdout.
xroad-metrics-collector update                     # Update Security Server list to MongoDB.
xroad-metrics-collector collect                    # Collect monitoring data from Security Server.
xroad-metrics-collector settings get mongodb.host  # Read a value from settings file and print to stdout

Above examples use the default settings file. To use another settings profile, you can use --profile flag:

xroad-metrics-collector --profile TEST list
xroad-metrics-collector --profile TEST update
xroad-metrics-collector --profile TEST collect

Cron settings

Default installation includes a cronjob in /etc/cron.d/xroad-metrics-collector-cron that runs collector every three hours. This job runs collector using default settings profile (/etc/xroad-metrics/collector/settings.yaml)

If you want to change the collector cronjob scheduling or settings profiles, edit the file e.g. with vi

vi /etc/cron.d/xroad-metrics-collector-cron

and make your changes. For example to run collector every six hours using settings profiles PROD and TEST:

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# m   h  dom mon dow  user       command
  15 */6  *   *   *   xroad-metrics  xroad-metrics-collector --profile PROD update && xroad-metrics-collector --profile PROD collect
  30 */6  *   *   *   xroad-metrics  xroad-metrics-collector --profile TEST update && xroad-metrics-collector --profile TEST collect

If collector is to be run only manually, comment out the default cron task:

# 20 */3  *   *   *   xroad-metrics      xroad-metrics-collector update && xroad-metrics-collector collect

Note about Indexing

Index build (see Database module, Index Creation might affect availability of cursor for long-running queries. Please review the need of active Collector module while running long-running queries in other modules.

Monitoring and Status

Logging

The settings for the log file in the settings file are the following:

xroad:
  instance: EXAMPLE

#  ...

logger:
  name: collector
  module: collector

  # Possible logging levels from least to most verbose are:
  # CRITICAL, FATAL, ERROR, WARNING, INFO, DEBUG
  level: INFO

  # Logs and heartbeat files are stored under these paths.
  # Also configure external log rotation and app monitoring accordingly.
  log-path: /var/log/xroad-metrics/collector/logs

The log file is written to log-path and log file name contains the X-Road instance name. The above example configuration would write logs to /var/log/xroad-metrics/collector/logs/log_collector_EXAMPLE.json.

Every log line includes:

  • "timestamp": timestamp in Unix format (epoch)
  • "local_timestamp": timestamp in local format '%Y-%m-%d %H:%M:%S %z'
  • "module": "collector"
  • "version": in form of "v${MINOR}.${MAJOR}"
  • "activity": possible values "collector_start", "collector_worker", "collector_end"
  • level: possible values "INFO", "WARNING", "ERROR"
  • msg: message

In case of "activity": "collector_end", the "msg" includes values separated by comma:

  • Total collected: number of Member's Security server from where the logs were collected successfully
  • Total error: number of Member's Security server from where the logs were not collected due to error
  • Total time: durations in the collection process in time format HH:MM:SS

The collector module log handler is compatible with the logrotate utility. To configure log rotation for the example setup above, create the file:

sudo vi /etc/logrotate.d/xroad-metrics-collector

and add the following content :

/var/log/xroad-metrics/collector/logs/log_collector_EXAMPLE.json {
    rotate 10
    size 2M
}

For further log rotation options, please refer to logrotate manual:

man logrotate

Heartbeat

The settings for the heartbeat file in the settings file are the following:

xroad:
  instance: EXAMPLE

#  ...

logger:
  #  ...
  heartbeat-path: /var/log/xroad-metrics/collector/heartbeat

The heartbeat file is written to heartbeat-path and heartbeat file name contains the X-Road instance name. The above example configuration would write logs to /var/log/xroad-metrics/collector/heartbeat/heartbeat_collector_EXAMPLE.json.

The heartbeat file consists last message of log file and status

  • status: possible values "FAILED", "SUCCEEDED"