Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: update eBPF/compilation instructions v5 #10840

Closed
wants to merge 4 commits into from
Closed

doc: update eBPF/compilation instructions v5 #10840

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
11fcea2
doc: update eBPF compilation instructions
Jan 16, 2024
3f35394
github-ci: remove gosu from installed packages
Apr 11, 2024
2d5a797
github-ci: add minimal and recommended build for Ubuntu and AlmaLinux
Apr 11, 2024
5bd4596
doc: port user install and build instruction from master-6.0.x
Jan 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 144 additions & 2 deletions .github/workflows/builds.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1348,6 +1348,87 @@ jobs:
exit 0
fi

almalinux-9-minimal-recommended-dependecies:
name: AlmaLinux 9 (Minimal/Recommended Build)
runs-on: ubuntu-latest
container: almalinux:9
needs: [prepare-deps]
steps:
# Cache Rust stuff.
- name: Cache cargo registry
uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
with:
path: ~/.cargo
key: ${{ github.job }}-cargo

- name: Cache RPMs
uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
with:
path: /var/cache/dnf
key: ${{ github.job }}-dnf
- run: echo "keepcache=1" >> /etc/dnf/dnf.conf

- name: Determine number of CPUs
run: echo CPUS=$(nproc --all) >> $GITHUB_ENV

- name: Enable required repositories
run: |
dnf -y update
dnf -y install dnf-plugins-core
dnf config-manager --set-enable crb

- name: Install git dependencies
run: |
dnf -y install \
sudo \
git \
libtool \
which

- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
- run: git config --global --add safe.directory /__w/suricata/suricata
- uses: actions/download-artifact@c850b930e6ba138125429b7e5c93fc707a7f8427
with:
name: prep
path: prep
- run: tar xf prep/libhtp.tar.gz
- run: tar xf prep/suricata-update.tar.gz
- run: tar xf prep/suricata-verify.tar.gz
- run: ./autogen.sh

- name: Install minimal dependencies
run: |
# install-guide-documentation tag start: Minimal RPM-based dependencies
sudo dnf install -y rustc cargo
cargo install --force cbindgen
echo 'export PATH+=":$HOME/.cargo/bin"' >> ~/.bashrc && export PATH+=":$HOME/.cargo/bin"
sudo dnf install -y gcc gcc-c++ jansson-devel libpcap-devel \
libyaml-devel make pcre2-devel zlib-devel
# install-guide-documentation tag end: Minimal RPM-based dependencies
echo "$HOME/.cargo/bin" >> $GITHUB_PATH

- run: CFLAGS="${DEFAULT_CFLAGS}" ./configure
- run: make -j ${{ env.CPUS }}
- run: ./src/suricata --build-info # check if we can run Suricata

# Recommended dependencies
- name: Install recommended dependencies
run: |
# install-guide-documentation tag start: Recommended RPM-based dependencies
sudo dnf install -y epel-release
sudo dnf install -y autoconf automake clang diffutils dpdk-devel \
file-devel hiredis-devel hyperscan-devel jansson-devel libbpf-devel \
libcap-ng-devel libevent-devel libmaxminddb-devel libnet-devel \
libnetfilter_queue-devel libnfnetlink-devel libpcap-devel \
libyaml-devel llvm-toolset lua-devel lz4-devel make \
nspr-devel nss-devel pcre2-devel pkgconfig python3-devel \
python3-sphinx python3-yaml zlib-devel
Comment on lines +1419 to +1425
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd remove clang, hiredis-devel, libeven-devel, libmaxminddb-devel, is llvm-toolset needed? nspr-devel, nss-devel can go as well. Python3-sphinx shouldn't be there for end-user dependencies.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang, hiredis-devel, libeven-devel, libmaxminddb-devel, is llvm-toolset needed? nspr-devel, nss-devel can go as well

The original "recommended dependencies" section contains these dependencies. While we don't pass any flags to configure, I though the purpose of the recommended step was to install all the recommended dependencies to be ready for most of the other configure flags that a user might want to enable.

If we e.g. remove clang, maxminddb, etc. then you could possibly just mention the minimal compilation dependency requirements and for other stuff let the user find the relevant sections of Suricata docs to install the required dependencies. (so e.g. if the user wants to include eBPF, the user searches for eBPF docs and from there, the dependencies and other instructions can be taken). This would mean that OISF is not favoring any other dependency except the required ones.

Is this something that you are aiming for?

# install-guide-documentation tag end: Recommended RPM-based dependencies

- run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-ebpf --enable-ebpf-build
- run: make -j ${{ env.CPUS }}
- run: ./src/suricata --build-info # check if we can run Suricata

ubuntu-22-04-cov-ut:
name: Ubuntu 22.04 (unittests coverage)
runs-on: ubuntu-latest
Expand Down Expand Up @@ -1999,6 +2080,69 @@ jobs:
- run: make -j ${{ env.CPUS }}
- run: ./src/suricata --build-info | grep -E "Netmap support:\s+yes"

ubuntu-22-04-minimal-recommended-build:
name: Ubuntu 22.04 (Minimal/Recommended Build)
needs: [prepare-deps, prepare-cbindgen]
runs-on: ubuntu-22.04
steps:
# Cache Rust stuff.
- name: Cache cargo registry
uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2
with:
path: ~/.cargo/registry
key: cargo-registry

- name: Determine number of CPUs
run: echo CPUS=$(nproc --all) >> $GITHUB_ENV

- name: Install git dependencies
run: |
sudo apt update
sudo apt -y install \
sudo \
git \
libtool

- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
- run: git config --global --add safe.directory /__w/suricata/suricata
- uses: actions/download-artifact@c850b930e6ba138125429b7e5c93fc707a7f8427
with:
name: prep
path: prep
- run: tar xf prep/libhtp.tar.gz
- run: tar xf prep/suricata-update.tar.gz
- run: tar xf prep/suricata-verify.tar.gz
- run: ./autogen.sh

- name: Install minimal dependencies
run: |
# install-guide-documentation tag start: Minimal dependencies
sudo apt -y install autoconf automake build-essential cargo \
cbindgen libjansson-dev libpcap-dev libpcre2-dev libtool \
libyaml-dev make pkg-config rustc zlib1g-dev
# install-guide-documentation tag end: Minimal dependencies

- run: CFLAGS="${DEFAULT_CFLAGS}" ./configure
- run: make -j ${{ env.CPUS }}
- run: ./src/suricata --build-info # check if we can run Suricata

# Recommended dependencies
- name: Install recommended dependencies
run: |
# install-guide-documentation tag start: Recommended dependencies
sudo apt -y install autoconf automake build-essential cargo \
cbindgen clang dpdk-dev libbpf-dev libgeoip-dev libhiredis-dev \
libjansson-dev liblua5.1-dev libmagic-dev libnetfilter-queue-dev \
libnetfilter-queue1 libnetfilter-log-dev libnetfilter-log1 \
libnfnetlink-dev libnfnetlink0 libpcap-dev libpcre2-dev \
libtool libyaml-dev make pkg-config python3 python3-dev \
python3-yaml rustc zlib1g-dev
# install-guide-documentation tag end: Recommended dependencies

- run: CFLAGS="${DEFAULT_CFLAGS}" ./configure --enable-ebpf --enable-ebpf-build
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think ebpf is recommended. If it is, it should be enabled by default.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to include recommended dependencies then it would make sense to continuously test if we can actually compile Suricata with the recommended dependencies for the recommended features. If we go with the minimal dependency only then this doesn't make sense indeed.

- run: make -j ${{ env.CPUS }}
- run: ./src/suricata --build-info # check if we can run Suricata

ubuntu-22-04-dpdk-build:
name: Ubuntu 22.04 (DPDK Build)
runs-on: ubuntu-22.04
Expand Down Expand Up @@ -2408,7 +2552,6 @@ jobs:
ccache \
curl \
git \
gosu \
jq \
libpcre2-dev \
libpcap-dev \
Expand Down Expand Up @@ -2491,7 +2634,6 @@ jobs:
ccache \
curl \
git \
gosu \
jq \
libpcre2-dev \
libpcap-dev \
Expand Down
1 change: 1 addition & 0 deletions Makefile.am
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ EXTRA_DIST = ChangeLog COPYING LICENSE suricata.yaml.in \
lua \
acsite.m4 \
scripts/generate-images.sh \
.github/workflows/builds.yml \
examples/plugins
SUBDIRS = $(HTP_DIR) rust src qa rules doc contrib etc python ebpf \
$(SURICATA_UPDATE_DIR)
Expand Down
11 changes: 5 additions & 6 deletions doc/userguide/capture-hardware/ebpf-xdp.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,16 +80,15 @@ Make sure you have ``clang`` (>=3.9) installed on the system ::

sudo apt install clang

Some i386 headers will also be needed as eBPF is not x86_64 and some included headers
are architecture specific ::

sudo apt install libc6-dev-i386 --no-install-recommends

libbpf
~~~~~~

Suricata uses libbpf to interact with eBPF and XDP ::

sudo apt install libbpf-dev

If the libbpf package is unavailable, it can be cloned from the repository ::

git clone https://github.com/libbpf/libbpf.git

Now, you can build and install the library ::
Expand All @@ -109,7 +108,7 @@ Compile and install Suricata
To get Suricata source, you can use the usual ::

git clone https://github.com/OISF/suricata.git
cd suricata && git clone https://github.com/OISF/libhtp.git -b 0.5.x
cd suricata && ./scripts/bundle.sh

./autogen.sh

Expand Down
120 changes: 48 additions & 72 deletions doc/userguide/install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,62 +61,27 @@ Common configure options

Enables `DPDK <https://www.dpdk.org/>`_ packet capture method.

Dependencies
^^^^^^^^^^^^

For Suricata's compilation you'll need the following libraries and their development headers installed::

libjansson, libpcap, libpcre2, libyaml, zlib

The following tools are required::

make gcc (or clang) pkg-config rustc cargo

Rust support::

rustc, cargo

Some distros don't provide or provide outdated Rust packages.
Rust can also be installed directly from the Rust project itself::

1) Install Rust https://www.rust-lang.org/en-US/install.html
2) Install cbindgen - if the cbindgen is not found in the repository
or the cbindgen version is lower than required, it can be
alternatively installed as: cargo install --force cbindgen
3) Make sure the cargo path is within your PATH environment
e.g. echo 'export PATH=”${PATH}:~/.cargo/bin”' >> ~/.bashrc
e.g. export PATH="${PATH}:/root/.cargo/bin"
Dependencies and compilation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Ubuntu/Debian
"""""""""""""

.. note:: The following instructions require ``sudo`` to be installed.

Minimal::

# Installed Rust and cargo as indicated above
sudo apt-get install build-essential git libjansson-dev libpcap-dev \
libpcre2-dev libtool libyaml-dev make pkg-config zlib1g-dev
# On most distros installing cbindgen with package manager should be enough
sudo apt-get install cbindgen # alternative: cargo install --force cbindgen
.. literalinclude:: ../../.github/workflows/builds.yml
:caption: Minimal dependencies for Ubuntu/Debian
:language: bash
:start-after: # install-guide-documentation tag start: Minimal dependencies
:end-before: # install-guide-documentation tag end: Minimal dependencies
:dedent: 10

Recommended::

# Installed Rust and cargo as indicated above
sudo apt-get install autoconf automake build-essential ccache clang curl git \
gosu jq libbpf-dev libcap-ng0 libcap-ng-dev libelf-dev \
libevent-dev libgeoip-dev libhiredis-dev libjansson-dev \
liblua5.1-dev libmagic-dev libnet1-dev libpcap-dev \
libpcre2-dev libtool libyaml-0-2 libyaml-dev m4 make \
pkg-config python3 python3-dev python3-yaml sudo zlib1g \
zlib1g-dev
cargo install --force cbindgen

Extra for iptables/nftables IPS integration::

sudo apt-get install libnetfilter-queue-dev libnetfilter-queue1 \
libnetfilter-log-dev libnetfilter-log1 \
libnfnetlink-dev libnfnetlink0
.. literalinclude:: ../../.github/workflows/builds.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm realizing now that we may not put these .github files in our release tarballs, so what happens when the .github dir isn't available?

Copy link
Contributor Author

@lukashino lukashino Apr 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that depends whether the docs are generated before or after the .github files are removed.
If the docs are generated prior to that then it is all fine because the literal text includes are already part of the docs.
If the docs are generated past .github folder removal then it will be missing and a warning will be generated during the doc build process.

I've tried to look into how we could substitute the missing text with a placeholder in case .github folder will be missing. Unfortunately, there is no such option atm. We could extend the literalinclude with our custom version.
Or we can again reverse the logic and use something similar approach but from .github. The build defined in .github will clone the Suricata repository, fetch the part of the documentation in between the tags and then execute it as a command.
But that is something we can think of if building the docs will be a problem in the end.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't make distcheck show this, then, if a problem exists?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would, that's why I had to add .github/workflows/builds.yml \ to Makefile.am
I was not sure if that also applies to the case when we package Suricata but if it does then that's even better :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember make distcheck being the most annoying and strict in terms of doc generation, when I was bringing the devguide into the userguide, and that we use that to check if Suri packages are generated properly. So I imagine that it would. 🤔 But not sure if we must double-check something else.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think our CI only does distcheck's from Git. A requirement of the dist is generating the documentation.

I bet if you re-did a distcheck from the output of a distcheck it might not work, as this ~/.github/ directory is not there.

So we should only rely on files that make it into the dist for the dist.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this PR updates the Makefile to include .github/workflows/builds.yml into the release.

:caption: Recommended dependencies for Ubuntu/Debian
:language: bash
:start-after: # install-guide-documentation tag start: Recommended dependencies
:end-before: # install-guide-documentation tag end: Recommended dependencies
:dedent: 10

CentOS, AlmaLinux, RockyLinux, Fedora, etc
""""""""""""""""""""""""""""""""""""""""""
Expand All @@ -129,45 +94,56 @@ one of the following ways::

sudo dnf -y update
sudo dnf -y install dnf-plugins-core
# AlmaLinux 8
# AlmaLinux 8 / RockyLinux 8
sudo dnf config-manager --set-enabled powertools
# AlmaLinux 9
# AlmaLinux 9 / RockyLinux 9
sudo dnf config-manager --set-enable crb
# Oracle Linux 8
sudo dnf config-manager --set-enable ol8_codeready_builder
# Oracle Linux 9
sudo dnf config-manager --set-enable ol9_codeready_builder

Minimal::

# Installed Rust and cargo as indicated above
sudo dnf install -y gcc gcc-c++ git jansson-devel libpcap-devel libtool \
libyaml-devel make pcre2-devel which zlib-devel
cargo install --force cbindgen
.. literalinclude:: ../../.github/workflows/builds.yml
:caption: Minimal dependencies for RPM-based distributions
:language: bash
:start-after: # install-guide-documentation tag start: Minimal RPM-based dependencies
:end-before: # install-guide-documentation tag end: Minimal RPM-based dependencies
:dedent: 10

Recommended::

# Installed Rust and cargo as indicated above
sudo dnf install -y autoconf automake diffutils file-devel gcc gcc-c++ git \
jansson-devel jq libcap-ng-devel libevent-devel \
libmaxminddb-devel libnet-devel libnetfilter_queue-devel \
libnfnetlink-devel libpcap-devel libtool libyaml-devel \
lua-devel lz4-devel make pcre2-devel pkgconfig \
python3-devel python3-sphinx python3-yaml sudo which \
zlib-devel
cargo install --force cbindgen
.. literalinclude:: ../../.github/workflows/builds.yml
:caption: Recommended dependencies for RPM-based distributions
:language: bash
:start-after: # install-guide-documentation tag start: Recommended RPM-based dependencies
:end-before: # install-guide-documentation tag end: Recommended RPM-based dependencies
:dedent: 10

Compilation
^^^^^^^^^^^
"""""""""""

Follow these steps from your Suricata directory::

./scripts/bundle.sh
./autogen.sh
./configure # you may want to add additional parameters here
# ./configure --help to get all available parameters
make -j8 # j is for paralleling, you may de/increase depending on your CPU
# j is for adding concurrency to make; the number indicates how much
# concurrency so choose a number that is suitable for your build system
make -j8
make install # to install your Suricata compiled binary
# make install-full - installs configuration and rulesets as well

Rust support
""""""""""""

Rust packages can be found in package managers but some distributions
don't provide Rust or provide outdated Rust packages.
In case of insufficient version you can install Rust directly
from the Rust project itself::

1) Install Rust https://www.rust-lang.org/en-US/install.html
2) Install cbindgen - if the cbindgen is not found in the repository
or the cbindgen version is lower than required, it can be
alternatively installed as: cargo install --force cbindgen
3) Make sure the cargo path is within your PATH environment
echo 'export PATH+=":~/.cargo/bin"' >> ~/.bashrc && export PATH+=":~/.cargo/bin"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds the path to the end, typically you want this before system ones.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, will change


Auto-Setup
^^^^^^^^^^
Expand Down
Loading