Detect rule hook/v9

[victorjulien]

SV_BRANCH=OISF/suricata-verify#2308

This is a start of the firewall specific ruleset as discussed in #12167:

It has a separate rule section for firewall rule files

  firewall-rule-path: /etc/suricata/firewall/
  firewall-rule-files:
    - fw.rules

Rules loaded in these files are different from regular ("threat detection") rules, in that:

1. that evaluated as they appear in the file (no reordering)
2. they require both action scope and "hook" to be explicit
3. things like ip-only are not available

The rules are currently quite restricted.

An example

pass:flow tcp:flow_start any any -> any 443 (flow:to_server; sid:1;)
drop:flow tcp:flow_start any any -> any any (sid:2;)

Wrt implementation, this PR keeps both firewall and "threat detection" rules in a single list, where the firewall rules go first.

Some option questions:

the behavior of "pass" - this is currently unmodified, meaning that a "firewall pass" would also skip threat detection. This doesn't seem to be what we want, as we'd like to say "accepted by firewall rules, now eval against threat detection". Perhaps there should be different classes of "pass" or a new firewall only action like "accept", that accepts but then still evals the threat detection rules.

I suppose this would need to be fine grained enough to distinguish between "pass but do threat detect" and "pass and skip any further inspection". Like how in iptables you'd have ACCEPT for straight accepting, and NFQUEUE for "accept if suricata agrees".

default policy drop - this is kind of tricky, as we have the different hooks. A hook would be equivalent to a "chain" in iptables I think, so a default drop should be per hook? I'm not really seeing this yet, esp if there are quite a few hooks for a single protocol like http.

There is also still some weirdness around inspection order.

Firewall:

pass:flow tcp:flow_start any any -> any 80 (flow:to_server; sid:1;)
drop:flow tcp:flow_start any any -> any any (sid:2;)
pass:flow tls:client_hello any any -> any 443 (tls.sni; content:"suricata.io"; sid:11;)
drop:flow tls:client_hello any any -> any any (sid:12;)

TD:

drop tcp any any -> any any (dsize:555; sid:33;)

Per packet, we would first inspect 1 and 2 as these are firewall packet hook rules. Then we would inspect 33, as this is a threat detect packet rule. Then we would eval the firewall app-tx rules. (ignore for now that this would require midstream with the client hello in the first packet for this to happen for a single packet)

I think the per hook order of 1. firewall, 2. threat detect is correct, but not sure about the case I described here.

So some more thinking is needed, would love some thoughts.

Oh for testing I added --firewall-rules-exclusive, which is like -S but loads a rulefile as a firewall rule.

Engine analysis also tries to be helpful

{
  "class": "firewall",
  "raw": "pass:packet tcp:flow_start any any -> any any (tcp.flags:!S; sid:10001;)",
  "id": 10001,
  "gid": 1,
  "rev": 0,
  "app_proto": "unknown",
  "requirements": [
    "tcp_flags_init_deinit",
    "real_pkt"
  ],
  "match_policy": {
    "actions": [
      "pass"
    ],
    "scope": "packet"
  },
  "type": "pkt",
  "flags": [
    "src_any",
    "dst_any",
    "sp_any",
    "dp_any",
    "noalert",
    "need_packet",
    "toserver",
    "toclient"
  ],
  "pkt_engines": [
    {
      "name": "packet",
      "is_mpm": false
    }
  ],
  "frame_engines": [],
  "lists": {
    "packet": {
      "matches": [
        {
          "name": "tcp.flags"
        }
      ]
    }
  }
}

(though I just realized it doesn't work with the exclusive commandline option yet)

[victorjulien]

cc @njlavigne

[codecov]

Codecov Report

Attention: Patch coverage is 85.36804% with 163 lines in your changes missing coverage. Please review.

Project coverage is 80.72%. Comparing base (80dbaac) to head (e67b1fa).
Report is 23 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12690      +/-   ##
==========================================
- Coverage   80.75%   80.72%   -0.03%     
==========================================
  Files         934      936       +2     
  Lines      259594   260010     +416     
==========================================
+ Hits       209634   209900     +266     
- Misses      49960    50110     +150     

Flag	Coverage Δ
fuzzcorpus	56.98% <62.31%> (-0.01%)	⬇️
livemode	19.56% <36.84%> (+0.19%)	⬆️
pcap	44.17% <41.97%> (+<0.01%)	⬆️
suricata-verify	63.59% <83.39%> (+0.07%)	⬆️
unittests	58.18% <53.50%> (-0.14%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

ERROR:

ERROR: QA failed on ASAN_TLPR1_cfg.

Pipeline 24930

[suricata-qa]

ERROR:

ERROR: QA failed on ASAN_TLPR1_cfg.

Pipeline 24933

[suricata-qa]

ERROR:

ERROR: QA failed on ASAN_TLPR1_cfg.

Pipeline 24934

[suricata-qa]

Information: QA ran without warnings.

Pipeline 24935

[suricata-qa]

Information: QA ran without warnings.

Pipeline 24936