From 3a848afc9cb21ae8322efda367fe66b99a109ce5 Mon Sep 17 00:00:00 2001 From: Anton Date: Fri, 24 Jan 2025 23:14:39 +0100 Subject: [PATCH] Docs for Validity 3.1 (#144) * articles written * docs fixes --- docs/entities/backuppoints.md | 1 + docs/entities/pollers.md | 7 ++ docs/entities/serializers.md | 2 +- docs/features/config_backup.md | 19 ++--- docs/features/custom_pollers.md | 36 +++++++-- docs/installation/installation.md | 2 +- docs/installation/netbox_compatibility.md | 19 ++++- docs/installation/plugin_settings.md | 98 ++++++++++++++++------- mkdocs.yml | 9 ++- 9 files changed, 145 insertions(+), 48 deletions(-) diff --git a/docs/entities/backuppoints.md b/docs/entities/backuppoints.md index 26cd610..9432346 100644 --- a/docs/entities/backuppoints.md +++ b/docs/entities/backuppoints.md @@ -19,6 +19,7 @@ If this param is `True`, backup will be performed each time respective Data Sour Defines protocol for uploading the Data Source. Available options are: + * Git (over http(s)) * Amazon S3 (or any other S3-compatible service) diff --git a/docs/entities/pollers.md b/docs/entities/pollers.md index 549b4a3..4161f00 100644 --- a/docs/entities/pollers.md +++ b/docs/entities/pollers.md @@ -64,3 +64,10 @@ There are 3 ways to bind a Poller to Device: * Set the Poller at **Device Type** level. Go to Device Type page at set the Poller via custom fields. This action applies this Poller to all the devices with this Device Type and overwrites the value from Manufacturer. * Set the Poller at the individual **Device** level. Go to Device page at set the Poller via custom fields. This action applies this Poller to one specific Device only and overwrites the values from Device Type and Manufacturer. + + +# Custom Pollers + +You can easily write your own polling backend, connect it to Validity and then use for polling. + +Read more about this feature in the [Custom Pollers](../features/custom_pollers.md) article. diff --git a/docs/entities/serializers.md b/docs/entities/serializers.md index e4bc48a..8fb3b40 100644 --- a/docs/entities/serializers.md +++ b/docs/entities/serializers.md @@ -139,7 +139,7 @@ ipv6: ``` ### XML -This method translates input XML-formatted text into Python dict using [xmltodict](https://github.com/martinblech/xmltodict) library. It is mainly used together with [Netconf commands](./commands.md#typenetconf). +This method translates input XML-formatted text into Python dict using [xmltodict](https://github.com/martinblech/xmltodict) library. It is mainly used together with [Netconf commands](./commands.md#type-netconf). **Input data:** ```xml diff --git a/docs/features/config_backup.md b/docs/features/config_backup.md index 5cc1a72..b07fc42 100644 --- a/docs/features/config_backup.md +++ b/docs/features/config_backup.md @@ -6,9 +6,9 @@ Validity supports device configurations backups to remote **Git** or **S3** serv 1. Follow the steps described in the [Quickstart: Polling](../quickstart_polling.md) article to set up all the entities required for device polling. -2. Polling is done when the Data Source is synced. In most cases it will be the default **Validity Polling** Data Source, but you are free to create your own one with the type *device_polling*. +2. Polling is done when the Data Source is synced. In most cases it will be the default **Validity Polling** Data Source, but you are free to create your own one with the type *device_polling*. -3. Create a [Backup Point](../entities/backuppoints.md) which describes a place where the Data Source with the configs have to be uploaded. +3. Create a [Backup Point](../entities/backuppoints.md) which describes a place where the Data Source with the configs has to be uploaded. Example using pynetbox: @@ -34,7 +34,7 @@ There are several options to trigger the backup process: * Press "Back Up" button on the Backup Point page. -* Specify `Backup After Sync: True` in the Backup Point settings. This will trigger the backup process each time the respective data source is being synced (no matter via button, API or somehow else). +* Specify `Backup After Sync: True` in the Backup Point settings. This will trigger the backup process each time the respective data source sync occurs (no matter via GUI button, API or somehow else). * Execute [RunTests](../entities/scripts.md#run-tests) script with `Sync Data Sources: True` option (`Backup After Sync: True` is required as well). @@ -43,13 +43,13 @@ There are several options to trigger the backup process: There are several options to provision periodic configuration backup process (e.g. daily, each 6 hours, etc): -1. Provision [RunTests](../entities/scripts.md#run-tests) script to execute on a regular basis using **Interval** script parameter. This will run everything at once: - * poll the devices (perform data source sync) - * execute compliance tests - * back up the data source +* Provision [RunTests](../entities/scripts.md#run-tests) script to start regularly using **Interval** script parameter. This will run everything at once: + 1. poll the devices (perform data source sync) + 2. execute compliance tests + 3. back up the data source -2. If you don't want to run the tests via scheduler and want backup only, you can create tiny [custom script](https://netboxlabs.com/docs/netbox/en/stable/customization/custom-scripts/) inside your NetBox and configure it to run periodically. +* If you don't want to run the tests via scheduler and want backup only, you can create tiny NetBox [Custom Script](https://netboxlabs.com/docs/netbox/en/stable/customization/custom-scripts/) inside your NetBox and configure it to run periodically. ```python from core.models import DataSource @@ -65,4 +65,5 @@ class SyncDataSource(Script): Execution of this script triggers the sync of the DataSource (and hence the backup process for bound Backup points with `Backup After Sync: True`). -3. Wait till periodic Data Source sync is implemented in the core NetBox. There is an [issue](https://github.com/netbox-community/netbox/issues/18287) for it, upvotes are appreciated. + +* Wait till periodic Data Source sync is implemented in the core NetBox. There is an [issue](https://github.com/netbox-community/netbox/issues/18287) for it, upvotes are appreciated. diff --git a/docs/features/custom_pollers.md b/docs/features/custom_pollers.md index 22e04b3..92f38dc 100644 --- a/docs/features/custom_pollers.md +++ b/docs/features/custom_pollers.md @@ -23,12 +23,16 @@ from validity.models import Command class ScrapliPoller(CustomPoller): + # this class/function will be supplied with poller parameters driver_factory = Scrapli - host_param_name = 'host' # Scrapli expects "host" param containing ip address of the device - driver_connect_method = 'open' # This driver method (if defined) will be called to open the connection. - driver_disconnect_method = 'close' # This driver method (if defined) will be called to gracefully close the connection. - - def poll_one_command(self, driver, command) -> str: + # Scrapli class expects "host" param containing ip address of the device + host_param_name = 'host' + # This driver method (if defined) will be called to open the connection. + driver_connect_method = 'open' + # This driver method (if defined) will be called to gracefully close the connection. + driver_disconnect_method = 'close' + + def poll_one_command(self, driver: Scrapli, command: Command) -> str: """ Arguments: driver - object returned by calling driver_factory, usually represents connection to a particular device @@ -64,9 +68,29 @@ PLUGIN_SETTINGS = { } ``` +??? info + The same setting may be defined via dict as well + ```python + # configuration.py + + from my_awesome_poller import ScrapliPoller + + PLUGIN_SETTINGS = { + 'validity': { + 'custom_pollers' : [ + 'class': ScrapliPoller, + 'name':'scrapli', + 'color':'pink', + 'command_types'=['CLI'], + ] + } + } + ``` + + PollerInfo parameters: -* **klass** - class inherited from `CustomPoller` +* **klass/class** - class inherited from `CustomPoller` * **name** - system name of the poller, must contain lowercase letters only * **verbose_name** - optional verbose name of the poller. Will be used in NetBox GUI * **color** - badge color used for "Connection Type" field in the GUI diff --git a/docs/installation/installation.md b/docs/installation/installation.md index 92efaf5..8b76e92 100644 --- a/docs/installation/installation.md +++ b/docs/installation/installation.md @@ -7,7 +7,7 @@ Validity is the [NetBox](https://netboxlabs.com/oss/netbox/) plugin. So, before | **Python** | **NetBox** | |------------|-----------------| -| >=3.10 | 3.7 | 4.0 | 4.1 | +| >=3.10 | 4.0 | 4.1 | 4.2 | ## Installation steps Once you have installed NetBox, you should follow these steps diff --git a/docs/installation/netbox_compatibility.md b/docs/installation/netbox_compatibility.md index 79c4b1e..12b2f69 100644 --- a/docs/installation/netbox_compatibility.md +++ b/docs/installation/netbox_compatibility.md @@ -26,7 +26,7 @@ In the table below you can find appropriate Validity version according to your N - + @@ -35,10 +35,11 @@ In the table below you can find appropriate Validity version according to your N + - + @@ -46,6 +47,7 @@ In the table below you can find appropriate Validity version according to your N + @@ -55,6 +57,7 @@ In the table below you can find appropriate Validity version according to your N + @@ -64,6 +67,7 @@ In the table below you can find appropriate Validity version according to your N + @@ -73,5 +77,16 @@ In the table below you can find appropriate Validity version according to your N + + + + + + + + + + +
NetBox
NetBox
v3.4v3.7 v4.0 v4.1v4.2
Validity

Validity

 v1.4 ✅ ✅
v2.2✅
v2.3✅ ✅
v3.0✅ ✅ ✅
v3.1✅✅✅
diff --git a/docs/installation/plugin_settings.md b/docs/installation/plugin_settings.md index 033b321..5469455 100644 --- a/docs/installation/plugin_settings.md +++ b/docs/installation/plugin_settings.md @@ -1,61 +1,85 @@ # Plugin Settings -Validity has some settings which can be changed through [PLUGINS_CONFIG](https://docs.netbox.dev/en/stable/plugins/#configure-plugin) variable inside your `configuration.py`. +Validity has some settings which can be changed through [PLUGINS_CONFIG](https://netboxlabs.com/docs/netbox/en/stable/plugins/installation/) variable inside your `configuration.py`. ## Settings -### result_batch_size +### **custom_pollers** -*Default:* `500` - -*Type:* `int` +*Type:* `list[validity.settings.PollerInfo]` -Execution of the Tests and producing Test Results is carried out in batches. As soon as each batch reaches its maximum size (specified via this variable) all the Test Results within a batch will be uploaded into a DB. +*Default:* `[]` +This setting allows to connect custom user-defined [Pollers](../entities/pollers.md) to Validity. -### store_reports +Read more about this feature in the [Custom Pollers](../features/custom_pollers.md) article. -*Default:* `5` -*Type:* `int` +### **custom_queues** -How many [Reports](../entities/results_and_reports.md#reports) should the system store. +*Type:* `dict[str, str]` -If the system creates a new Report and the overall reports count exceeds *store_reports*, then the oldest exceeding report(s) will be deleted. +This settings defines custom RQ queue names for Validity scripts. It may be useful if you want to move script execution into a separate queue (e.g. handled by separate set of workers) !!! note - Test Results bound to the Report will be deleted as well. + Default RQ worker serves 3 queues only: `high`, `default` and `low`. You have to reconfigure your RQ worker (or just start another one in parallel) in case of choosing a different queue name. + For instance, if you have two custom queues `my_q1` and `my_q2` and want a separate worker for them, then the worker has to be started with these queue names as ths parameters: -### polling_threads + `./manage.py rqworker my_q1 my_q2` -*Default:* `500` -*Type:* `int` +Here are the custom queues which may be defined via this setting: -Validity uses threads to perform device polling. This setting defines the upper limit of these threads number. If you have extremely large amount of devices (e.g. 100 000), you may want to increase this number to speed up the overall polling process. +| Queue Name | Description | Default value | +|---|---|---| +| runtests | Queue for Run Tests script | default | +| backup | Queue for backing up individual Backup Points (Back Up button) | default | +!!! warning + The `runtests_queue` setting is **deprecated since version 3.1**. Use `custom_queues.runtests` instead -### runtests_queue -*Default:* `"default"` +### **integrations** -*Type:* `str` +*Type:* `dict[str, dict]` -RQ queue name for running the tests. May be useful if you want to move tests execution into a separate queue (e.g. handled by separate set of workers) +*Default:* -!!! note - Default RQ worker serves 3 queues only: `high`, `default` and `low`. You have to reconfigure your RQ worker (or just start another one in parallel) in case of choosing a different queue name. +```python +{ + 'git': {'author': 'netbox-validity', 'email': 'validity@netbox.local'} + 'S3': {'threads': 10} +} +``` + +This setting is responsible for various integrations with third-party services, mostly for [Config Backup](../features/config_backup.md) feature. + +`threads` param is responsible for the level of parallelism (number of threads used) for multiple files uploading to S3. + + +### **polling_threads** + +*Default:* `500` - For instance, if your runtests_queue is set to `my_queue`, then RQ worker has to be started with this queue name as a parameter: +*Type:* `int` + +Validity uses threads to perform device polling. This setting defines the upper limit of these threads number. If you have extremely large amount of devices (e.g. 100 000), you may want to increase this number to speed up the overall polling process. + + +### **result_batch_size** + +*Default:* `500` + +*Type:* `int` - `./manage.py rqworker my_queue` +Execution of the Tests and producing Test Results is carried out in batches. As soon as each batch reaches its maximum size (specified via this variable) all the Test Results within a batch will be uploaded into database. -### script_timeouts +### **script_timeouts** -*Type:* `dict` +*Type:* `dict[str, str | int]` This setting defines the timeouts for RQ jobs started by Validity (e.g. for running the tests). Timeout defines the maximum amount of time some job can run. If a job exceeds the timeout, it gets terminated. @@ -66,11 +90,28 @@ As was mentioned in the [corresponding article](../entities/scripts.md#stages), | runtests_split | 10m | | runtests_apply | 30m | | runtests_combine | 10m | +| backup | 10m | +### **store_reports** + +*Default:* `5` + +*Type:* `int` + +How many [Reports](../entities/results_and_reports.md#reports) should the system store. + +If the system creates a new Report and the overall reports count exceeds *store_reports*, then the oldest exceeding report(s) will be deleted. + +!!! warning + Test Results bound to the Report will be deleted as well. + ## Settings Example +!!! note + The following example does NOT represent the recommended setting values. + Here is the full example of Validity settings: ```python @@ -78,10 +119,13 @@ Here is the full example of Validity settings: PLUGINS_CONFIG = { 'validity': { + 'integrations': { + 'git': {'author': 'Anton'} + }, 'result_batch_size': 300, 'store_reports': 7, 'polling_threads': 100, - 'runtests_queue': 'validity_tests', + 'custom_queues': {"runtests": "runtests_q"}, 'script_timeouts': { 'runtests_split': '15m', 'runtests_apply': '1h', diff --git a/mkdocs.yml b/mkdocs.yml index a3f7c81..b249aa3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,6 +1,9 @@ site_name: Validity repo_url: https://github.com/amyasnikov/validity/ -theme: material +theme: + name: material + icon: + logo: material/checkbox-marked-circle-outline nav: - Intro: index.md - Installation: @@ -26,7 +29,8 @@ nav: - How to work with Device State: features/state.md - Dynamic Pairs: features/dynamic_pairs.md - Webhooks: features/webhooks.md - - Config backup: features/config_backup.md + - Custom Pollers: features/custom_pollers.md + - Config Backup: features/config_backup.md markdown_extensions: - pymdownx.highlight: @@ -34,6 +38,7 @@ markdown_extensions: line_spans: __span pygments_lang_class: true - admonition + - pymdownx.details - pymdownx.inlinehilite - pymdownx.snippets - pymdownx.superfences