hyperscan: add caching mechanism for hyperscan contexts v10

[lukashino]

Followup of #12048

Cache Hyperscan serialized databases to disk to prevent compilation of the same databases when Suricata is run again with the same ruleset.
Hyperscan binary files are stored per rulegroup in the designated folder, by default in the cached library folder.
Since caching is per signature group heads, some chunk of the ruleset can change and it still can reuse part of
the unchanged signature groups.

Loading fresh ET Open ruleset: 19 seconds
Loading cached ET Open ruleset: 07 seconds

Link to ticket: https://redmine.openinfosecfoundation.org/issues/7170

Describe changes:
v10:

• the default path to the caching directory is not HS tied and is code-embedded through a macro
• minor user/code adjustments
• commit cleanups
• rebased

v9:

• rebase
• split HS MPM codebase into multiple files - core functions, Hyperscan MPM functions, HS caching functions
• recommitted the code
• minor tweaks from the previous PR, docs update
• made DetectEngineMpmCachingEnabled private (possible also for DetectEngineMpmCachingGetPath), but checkout v8 thread - decision needed
• some discussion threads in v8 can/should be brought up again and be decided

v7: (v6 was private)

• fix docs and add ticket number to the commit
• fix privilege drop issue, files are created after privilege drop, tested also rule reload - it worked fine
• refactor the util-mpm-hs code, primarily prepare function
• rebase

v5:

• rebased
• commit message update
• docs update

v4:

• rebased
• changed the default caching directory to somewhere /var/lib/suricata/cache/hs
• custom cache directory path option added
• docs added
• the default settings changed - enabled on the config generation, disabled when the option is not present in the config

v3

• rebased
• MPM caching is still left on by default.

v2

• improved styling to follow Suricata code styleguide
• increased cache file name length from 10 to 20 characters
• cache file name is a hash of the patterns - now only HS relevant fields are hashed - as long as the group of patterns itself is not changed then it is reused
• minor refactors
• added a safe variant of littlehash2 function
• added suricata.yaml option to enable/disable caching
• changed the storage location to the configured logging directory

v1

• initial work to cache and load Hyperscan databases from the disk

[codecov]

Codecov Report

Attention: Patch coverage is 30.74324% with 410 lines in your changes missing coverage. Please review.

Project coverage is 80.49%. Comparing base (05853fb) to head (8f01d2e).
Report is 192 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12394      +/-   ##
==========================================
- Coverage   82.46%   80.49%   -1.98%     
==========================================
  Files         914      917       +3     
  Lines      258316   258854     +538     
==========================================
- Hits       213030   208370    -4660     
- Misses      45286    50484    +5198     

Flag	Coverage Δ
fuzzcorpus	56.79% <10.24%> (-3.59%)	⬇️
livemode	19.38% <11.21%> (-0.01%)	⬇️
pcap	44.23% <10.73%> (-0.11%)	⬇️
suricata-verify	63.17% <11.70%> (-0.10%)	⬇️
unittests	58.43% <29.89%> (+0.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

Information: QA ran without warnings.

Pipeline 24210

[victorjulien]

How will this behave around rule reloads and testing? The recommended way with suricata-update is to run suri on the new rules with -T before issuing the reload to the running suricata. What behavior will there be around caching here? Also if the -T run fails, what happens?

[lukashino]

Reply can be found in #12658