improve CPU cores affinity to NUMA nodes v9.5

[lukashino]

Followup of #12369

Redmine ticket:
https://redmine.openinfosecfoundation.org/issues/7036
https://redmine.openinfosecfoundation.org/issues/2321

This work allows more precise thread assignment and it is done either

• automatically - by picking assigned cores of the same NUMA locality as the interface from the worker-cpu-list
• manually - you can specify interface-specific settings in the threading section

This works with AF-PACKET and DPDK (and should with other capture modes as well). The primary target is workers runmode but it also works with autofp receive threads.

To try this PR:

1. compile Suri from this branch
2. Modify the newly generated suricata.yaml according to your setup.
3. Run as usual.

Feedback

Feedback was given in the previous PRs:

• Corey comments: improve CPU cores affinity to NUMA nodes v8 #12336, asks about extra logic, in improve CPU cores affinity to NUMA nodes v9 #12359 I addressed the comments
• Jeff and Andreas gave their feedback in improve CPU cores affinity to NUMA nodes v9.4 #12369
• Jason commented on the autopin naming

Feedback sum up:

• Corey asked about extra logic to express NUMA affinity, e.g. - adding an exclusion logic through variables e.g. worker-cpu-set: [ "!management-cpu-set" ] - that I view as an extension of this work
• Jeff asked about on a similar note - to define exclusion sets for autopinning - similarly, I view this as an extra feature

Describe changes:

v9.5:

• addressed Jeff's docs comments
• added a "Feedback" section to the pull request description where I summed up the received feedback
• rebased

v9.4:

• scan-build fix - reorder ThreadAffinity structure members

v9:

• docs update per PRv8 comments
• CPU affinity fix for af-packet when autopinning is enabled and threads are oversubscribed
• addressed PR comments

v8:

• interface-specific settings now don't require hwloc dependency
• CPU affinity refactoring changes segregated to a separate commit
• added docs - upgrade docs and general threading docs
• a little change in a title

v7:

• hwloc is now an optional dependency - it can be enabled with --enable-hwloc in the configure step,
• autopinning / autooptimizing is configurable from threading.autopin suri.yaml option,
• making CI happy (and with that fixing a few bugs)

v6

• support for autofp mode supported - receive-cpu-set can now contain per interface CPU affinity settings - here comes a question - should worker-cpu-set in autofp be somehow configurable to the receive-cpu-set threads?
• YAML node "per-iface" changed to "interface-specific-cpu-set"
• list form of (management/worker/...)-cpu-set was changed to simple tree nodes
• cpu/prio/mode properties of (management/worker/...)-cpu-set nodes moved one YAML layer up
• DPDK (net_bonding / PCIe address / ... ) / AFP interfaces are supported

On YAML changes:

CPU affinity section in suricata.yaml has been changed to (see suricata.yaml.in for the full picture):

threading:
  cpu-affinity:
    worker-cpu-set:
      cpu: [ "all" ]
      mode: "exclusive"
      interface-specific-cpu-set:
        - interface: "net_bonding0" # 0000:3b:00.0 # net_bonding0 # ens1f0
          cpu: [ "all" ]
        - interface: "net_bonding1"
          cpu: [ "all" ]

Below is the reasoning for it.

When starting this work I wanted to follow the same structure as CPU set nodes that is:

threading:
  cpu-affinity:
    - worker-cpu-set:
        cpu: [ "all" ]
        mode: "exclusive"

converted to JSON it looks like:

{
    "threading": {
        "cpu-affinity": [
            {
                "worker-cpu-set": {
                    "cpu": [
                        "all"
                    ],
                    "mode": "exclusive"
                }
            }
        ]
    }
}

So worker-cpu-set is an object by itself, it holds the key that is named "worker-cpu-set" and only that contains its children properties (CPU, prio, etc.).

But when adding the interface-specific node I found out that having the interface names directly as the keys for the object storing the properties (CPU, prio, etc.) can change its name to something else, in this case to net-bonding0.

threading:
  cpu-affinity:
    - worker-cpu-set:
        cpu: [ "all" ]
        mode: "exclusive"
        interface-specific-cpu-set:
          - net_bonding0:
              cpu: [ "all" ]
          - net_bonding1:
              cpu: [ "all" ]

So adding it as a value:

threading:
  cpu-affinity:
    - worker-cpu-set:
        cpu: [ "all" ]
        mode: "exclusive"
        interface-specific-cpu-set:
          - interface: net_bonding0
              cpu: [ "all" ]
          - interface: net_bonding1
              cpu: [ "all" ]

yields an invalid YAML construct.

Therefore, for interface-specific CPU affinity settings, I ended up putting it on the same level as the interface name itself. This follows the same structure as is established in e.g. capture-mode interface definition (dpdk.interfaces, af-packet).
Note: In the matter of this I transferred the base *-cpu-set nodes from the list items as part of the Ticket 2321.
The CPU affinity interface-specific design I ended up with is:

threading:
  cpu-affinity:
    worker-cpu-set:
          cpu: [ "all" ]
          mode: "exclusive"
          interface-specific-cpu-set:
            - interface: "net_bonding0" # 0000:3b:00.0 # net_bonding0 # ens1f0
              cpu: [ "all" ]
            - interface: "net_bonding1"
              cpu: [ "all" ]

JSON visualization of this:

{
    "threading": {
        "cpu-affinity": {
            "worker-cpu-set": {
                "cpu": [
                    "all"
                ],
                "mode": "exclusive",
                "interface-specific-cpu-set": [
                    {
                        "interface": "net_bonding0",
                        "cpu": [
                            "all"
                        ]
                    },
                    {
                        "interface": "net_bonding1",
                        "cpu": [
                            "all"
                        ]
                    }
                ]
            }
        }
    }
}

This leaves the base {worker,receive,management}-cpu-set nodes in the original setting and as we know it. The internal code can handle this because the names of these are known beforehand and can "step down" to the children level to inspect cpu-affinity properties. The dynamic part - interface-specific-cpu-set contains all properties on the same level.

Other unsuccessful YAML designs

Specifying the NUMA affinity in the individual capture modes

Example

dpdk:
  - interface: "3b:00.0"
    threads: [ 2, 4, 6, 8]

But management / other threads would still be required to configure elsewhere. I liked the proposed approach more because all affinity-related settings are centralized.

Not having a list nodes in the YAML

threading:
  worker-cpu-set:
    net_bonding0:
      cpu: [ 1, 3, 5 ]

YAML library changes net_bonding to net-bonding when YAML is loaded - so this is a nogo.

Have interface-related properties as children

threading:
  worker-cpu-set:
    - interface: net_bonding0
        cpu: [ 1, 3, 5 ]

This is a disallowed construct in YAML, so another nogo. A similar pattern is currently with *-cpu-set nodes but the different here is that the parent list item has a value (in this case net_bonding0). That is what is causing problems. Interface related therefore need to right under "interface".

PRO TIP: A good way to visualize different YAML structures is by using YAML to JSON converters. Currently it creates object like:

[
  {
    "interface": "3b:00.0",
    "mode": "exclusive",
  }
]

[codecov]

Codecov Report

Attention: Patch coverage is 2.59067% with 376 lines in your changes missing coverage. Please review.

Project coverage is 80.17%. Comparing base (3bc2a14) to head (1582ecb).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12656      +/-   ##
==========================================
- Coverage   80.77%   80.17%   -0.61%     
==========================================
  Files         932      933       +1     
  Lines      259517   259785     +268     
==========================================
- Hits       209629   208283    -1346     
- Misses      49888    51502    +1614     

Flag	Coverage Δ
fuzzcorpus	56.90% <0.00%> (-0.09%)	⬇️
livemode	18.29% <2.59%> (-1.09%)	⬇️
pcap	44.10% <0.30%> (-0.06%)	⬇️
suricata-verify	63.42% <0.30%> (-0.10%)	⬇️
unittests	58.27% <0.00%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

WARNING:

field	baseline	test	%
	TREX_GENERIC_suri

Pipeline 24861