-
-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
0b51fc0
commit 89e9f68
Showing
16 changed files
with
284 additions
and
124 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,25 +1,47 @@ | ||
| # Custom Scripts | ||
| # Scripts | ||
|
|
||
| Validity is shipped together with some [Custom Scripts](https://docs.netbox.dev/en/stable/customization/custom-scripts/). | ||
| **Scripts** is the way Validity executes long-running tasks. Previously Validity used [NetBox Custom Scripts](https://netboxlabs.com/docs/netbox/en/stable/customization/custom-scripts/) feature for this purpose, but since **release 3.0.0** Validity uses [RQ](https://python-rq.org/) directly to execute its long-running jobs (out of habit, still called "scripts"). | ||
|
|
||
| The scripts are available under `Customization > Scripts` GUI menu. | ||
| ## Run Tests | ||
|
|
||
| ## Run Compliance Tests | ||
| This script executes Compliance Tests and creates Report and Test Result instances. | ||
|
|
||
| This script executes Compliance Tests and creates Test Result instances. | ||
| The script also performs a bunch of auxiliary actions such as: | ||
|
|
||
| This script also deletes the old Test Results and Reports if they exceed the maximum number from [settings](../plugin_settings.md). | ||
| * Related Data Sources Sync (if requested from user) | ||
| * Deletion of old Reports and Tests Results (see [store_reports](../installation/plugin_settings.md#store_reports) setting) | ||
|
|
||
| The script may generate a lot of DB queries. To spread the queries over time you can adjust [sleep_between_tests](../plugin_settings.md#sleep_between_tests) setting. | ||
|
|
||
| #### Params | ||
| ### Parallel Tests Execution | ||
|
|
||
| Validity allows splitting Tests Execution between multiple RQ workers. In this case each RQ worker executes its own portion of the work reducing the overall time required to process all amount of Tests. | ||
|
|
||
| Tests Execution is divided between multiple workers on a per-device basis. It means that all the Tests for one particular Device will always be executed on one single worker. | ||
|
|
||
| For example if you have 100 devices and want to execute the Tests for them on 3 workers in parallel, just choose `workers_num=3`, it will lead to two of the workers serving 33 devices each and the last one serving 34. | ||
|
|
||
| !!! note | ||
| It's the responsibility of NetBox administrator to spawn multiple RQ workers. In most of the deployments there is only one worker by default. | ||
|
|
||
|
|
||
| ### Execution Params | ||
|
|
||
| | **Param** | **API Param** | **Description** | | ||
| |-----------------------------|---------------|-------------------------------------------------------------------| | ||
| | Sync Data Sources | sync_datasources | Sync all Data Sources which are bound to Devices<br/> participating in the script execution| | ||
| | Make Compliance<br/>Report | make_report | Create Report together with Test Results | | ||
| | Sync Data Sources | sync_datasources | Sync all Data Sources which are bound to Devices participating in the script execution| | ||
| | Specific Selectors | selectors | Run the tests only for a limited number of selectors | | ||
| | Specific Devices | devices | Run the tests only for a limited number of devices. | | ||
| | Specific Test Tags | test_tags | Run only those tests which have at least one of<br/> the specified tags| | ||
| | Explanation Verbosity<br/>Level | explanation_verbosity | **0** - No explanation at all.<br/>**1** - Explanation of the calculation steps<br/>**2** - the same as **1** plus deepdiff value in case of<br/> comparisons| | ||
| | Override DataSource | override_datasource| Ignore Data Sources bound to Devices and use<br/> this one instead. It may be useful if you want to<br/> use **Validity Polling** Data Source just to run some<br/> operational tests only for now.| | ||
| | Specific Test Tags | test_tags | Run only those tests which have at least one of the specified tags| | ||
| | Explanation Verbosity Level | explanation_verbosity | **0** - No explanation at all.**1** - Explanation of the calculation steps**2** - the same as **1** plus deepdiff value in case of comparisons| | ||
| | Number of Workers | workers_num | Number of RQ workers to split test execution between them | | ||
| | Override DataSource | overriding_datasource| Ignore Data Sources bound to Devices and use this one instead. It may be useful if you want to use **Validity Polling** Data Source just to run some operational tests only for now.| | ||
|
|
||
|
|
||
| ### Stages | ||
| Internally Run Tests script has 3 stages: | ||
|
|
||
| * **split**: synchronizes Data Sources (if requested by user) and splits test execution into splices | ||
| * **apply**: Executes portion of tests assigned to it at split stage and saves the results to database. Only this stage can be executed on multiple workers in parallel | ||
| * **combine**: collects the overall statistics, fires webhook (if configured) | ||
|
|
||
| Each stage has its own timeout which can be adjusted in [Plugin Settings](../installation/plugin_settings.md#script_timeouts) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| .md-grid { | ||
| max-width: 1470px; | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| # NetBox Compatibility | ||
|
|
||
| Each Validity version supports 3 NetBox releases: the current one (at the moment of the release) plus two previous ones. | ||
|
|
||
| In the table below you can find appropriate Validity version according to your NetBox version. | ||
|
|
||
| !!! info | ||
| Only the most appropriate Validity versions are listed in the table. For example, v2.0 is not listed in favor of v2.2 (v2.0 and v2.2 support the same set of NetBox versions, but v2.2 has more features). | ||
|
|
||
|
|
||
|
|
||
| #### NetBox Compatibility Table | ||
|
|
||
| <style type="text/css"> | ||
| .tg {border-collapse:collapse;border-color:#9ABAD9;border-spacing:0;} | ||
| .tg td{background-color:#EBF5FF;border-color:#9ABAD9;border-style:solid;border-width:1px;color:#444; | ||
| font-family:Arial, sans-serif;font-size:14px;overflow:hidden;padding:10px 5px;word-break:normal;} | ||
| .tg th{background-color:#409cff;border-color:#9ABAD9;border-style:solid;border-width:1px;color:#fff; | ||
| font-family:Arial, sans-serif;font-size:14px;font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;} | ||
| .tg .tg-r82u{background-color:#ebf5ff;border-color:#330001;text-align:center;vertical-align:top} | ||
| .tg .tg-c6ba{border-color:#330001;font-weight:bold;text-align:center;vertical-align:top} | ||
| .tg .tg-5gjp{background-color:#ebf5ff;border-color:#330001;color:#444444;font-weight:bold;text-align:center;vertical-align:top} | ||
| .tg .tg-5iin{background-color:#409cff;border-color:#330001;color:#ebf5ff;font-weight:bold;text-align:center;vertical-align:top} | ||
| .tg .tg-8g55{border-color:#330001;text-align:center;vertical-align:top} | ||
| </style> | ||
| <table class="tg"><thead> | ||
| <tr> | ||
| <th class="tg-r82u" colspan="2" rowspan="2"></th> | ||
| <th class="tg-c6ba" colspan="6">NetBox<br></th> | ||
| </tr> | ||
| <tr> | ||
| <th class="tg-5gjp">v3.4</th> | ||
| <th class="tg-5gjp">v3.5</th> | ||
| <th class="tg-5gjp">v3.6</th> | ||
| <th class="tg-5gjp">v3.7</th> | ||
| <th class="tg-5gjp">v4.0</th> | ||
| <th class="tg-5gjp">v4.1</th> | ||
| </tr></thead> | ||
| <tbody> | ||
| <tr> | ||
| <td class="tg-5iin" rowspan="4">Validity<br><br></td> | ||
| <td class="tg-c6ba">v1.4</td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55"></td> | ||
| </tr> | ||
| <tr> | ||
| <td class="tg-c6ba">v2.2</td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55"></td> | ||
| </tr> | ||
| <tr> | ||
| <td class="tg-c6ba">v2.3</td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55"></td> | ||
| </tr> | ||
| <tr> | ||
| <td class="tg-c6ba">v3.0</td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55"></td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55">✅</td> | ||
| <td class="tg-8g55">✅</td> | ||
| </tr> | ||
| </tbody></table> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| # Plugin Settings | ||
|
|
||
|
|
||
| Validity has some settings which can be changed through [PLUGINS_CONFIG](https://docs.netbox.dev/en/stable/plugins/#configure-plugin) variable inside your `configuration.py`. | ||
|
|
||
| ## Settings | ||
|
|
||
| ### result_batch_size | ||
|
|
||
| *Default:* `500` | ||
|
|
||
| *Type:* `int` | ||
|
|
||
| Execution of the Tests and producing Test Results is carried out in batches. As soon as each batch reaches its maximum size (specified via this variable) all the Test Results within a batch will be uploaded into a DB. | ||
|
|
||
|
|
||
| ### store_reports | ||
|
|
||
| *Default:* `5` | ||
|
|
||
| *Type:* `int` | ||
|
|
||
| How many [Reports](../entities/results_and_reports.md#reports) should the system store. | ||
|
|
||
| If the system creates a new Report and the overall reports count exceeds *store_reports*, then the oldest exceeding report(s) will be deleted. | ||
|
|
||
| !!! note | ||
| Test Results bound to the Report will be deleted as well. | ||
|
|
||
|
|
||
| ### polling_threads | ||
|
|
||
| *Default:* `500` | ||
|
|
||
| *Type:* `int` | ||
|
|
||
| Validity uses threads to perform device polling. This setting defines the upper limit of these threads number. If you have extremely large amount of devices (e.g. 100 000), you may want to increase this number to speed up the overall polling process. | ||
|
|
||
|
|
||
| ### runtests_queue | ||
|
|
||
| *Default:* `"default"` | ||
|
|
||
| *Type:* `str` | ||
|
|
||
| RQ queue name for running the tests. May be useful if you want to move tests execution into separate queue (e.g. handled by separate set of workers) | ||
|
|
||
|
|
||
| ### script_timeouts | ||
|
|
||
| *Type:* `dict` | ||
|
|
||
| This setting defines the timeouts for RQ jobs started by Validity (e.g. for running the tests). Timeout defines the maximum amount of time some job can run. If a job exceeds the timeout, it gets terminated. | ||
|
|
||
| As was mentioned in the [corresponding article](../entities/scripts.md#stages), **Run Tests** encompass 3 stages (split, apply, combine) where each stage has its own timeout. | ||
|
|
||
| | Timeout | Default Value | | ||
| |------------------|---------------| | ||
| | runtests_split | 10m | | ||
| | runtests_apply | 30m | | ||
| | runtests_combine | 10m | | ||
|
|
||
|
|
||
|
|
||
| ## Settings Example | ||
|
|
||
| Here is the full example of Validity settings: | ||
|
|
||
| ```python | ||
| # configuration.py | ||
|
|
||
| PLUGINS_CONFIG = { | ||
| 'validity': { | ||
| 'result_batch_size': 300, | ||
| 'store_reports': 7, | ||
| 'polling_threads': 100, | ||
| 'runtests_queue': 'validity_tests', | ||
| 'script_timeouts': { | ||
| 'runtests_split': '15m', | ||
| 'runtests_apply': '1h', | ||
| 'runtests_combine': '5m', | ||
| }, | ||
| }, | ||
| # other plugins configuration | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # Upgrading to a new Validity release | ||
|
|
||
| Upgrade process is pretty straightforward: | ||
|
|
||
| * Make sure your NetBox version [is compatible](./netbox_compatibility.md) with the new Validity version | ||
|
|
||
| * Update the package version through pip | ||
|
|
||
| ``` | ||
| pip install --upgrade netbox-validity | ||
| ``` | ||
|
|
||
| * Run `migrate` and `collectstatic` management commands | ||
|
|
||
| ``` | ||
| ./manage.py migrate | ||
| ./manage.py collectstatic | ||
| ``` | ||
|
|
||
|
|
||
| ## Major release upgrade | ||
|
|
||
| Major version upgrade (e.g. v2.x -> v3.x) may require some additional rules: | ||
|
|
||
| * Don't skip major version upgrades (e.g. don't upgrade 1.4.1 directly to 3.0.0) | ||
| * Before upgrading major version number, install the latest minor version first | ||
|
|
||
| For example, you want to upgrade from 2.1.1 to 3.0.0. To do this you have to | ||
|
|
||
| * install 2.3.3 (the latest available v2 release) first according to the procedure described above | ||
| * install 3.0.0 |
Oops, something went wrong.