Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[backport v1.0] pkg/sensors: reduce stack trace map memory footprint #2549

Merged
merged 1 commit into from
Jul 1, 2024

Conversation

mtardy
Copy link
Member

@mtardy mtardy commented Jun 13, 2024

[ upstream commit 22510d9 ]
This is a slightly adapted version because #2145 was merged after v1.0

We stopped on a stack trace map that has a max_entries of 32768, which is 64 bits pointers * PERF_MAX_STACK_DEPTH (which is fixed at 127 for now), so 127*64/8=1016 bytes per entry + it's key_size of 32 bits (4 bytes) so 1020 bytes per entry. So 1020 * 32768 = 33,423,360 bytes. From bpftool, this map has a total bytes_memlock of 34,079,040 bytes. So for each stack trace map we load, we had 34MB of kernel memory, and it happened to be loaded many times when we were loading any tracing policy.

Since the map is used by the generic program, the loader will allocate the memory needed for the map even if we don't create a reference from the agent side and create an anonymous map. So we end up allocating a small map of max_entries 1 by default and resize it when the tracing policy actually specifies a matchAction containing a kernelStackTrace or userStackTrace to true. This should drastically reduce the memory footprint of this feature when it's unused.

Reduce the kernel memory footprint (accounted by the cgroup memory controller) of the stack trace feature when unused.

@mtardy mtardy added kind/backport This PR provides functionality previously merged into master. release-note/bug This PR fixes an issue in a previous release of Tetragon. labels Jun 13, 2024
@mtardy mtardy requested a review from olsajiri June 13, 2024 09:20
@mtardy mtardy requested a review from a team as a code owner June 13, 2024 09:20
@mtardy mtardy force-pushed the pr/mtardy/backport-v1.0-stacktrace-memory branch from a4a642d to 352392e Compare June 19, 2024 11:17
[ upstream commit 22510d9 ]

[ This is a slightly adapted version since #2145 was merged after v1.0 ]

We stopped on a stack trace map that has a max_entries of 32768, which
is 64 bits pointers * PERF_MAX_STACK_DEPTH (which is fixed at 127 for
now), so 127*64/8=1016 bytes per entry + it's key_size of 32 bits (4
bytes) so 1020 bytes per entry. So 1020 * 32768 = 33,423,360 bytes.
From bpftool, this map has a total bytes_memlock of 34,079,040 bytes.
So for each stack trace map we load, we had 34MB of kernel memory, and
it happened to be loaded many times when we were loading any tracing
policy.

Since the map is used by the generic program, the loader will allocate
the memory needed for the map even if we don't create a reference from
the agent side and create an anonymous map. So we end up allocating a
small map of max_entries 1 by default and resize it when the tracing
policy actually specifies a matchAction containing a kernelStackTrace or
userStackTrace to true. This should drastically reduce the memory
footprint of this feature when it's unused.

Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>
@mtardy mtardy force-pushed the pr/mtardy/backport-v1.0-stacktrace-memory branch from 352392e to e2205a4 Compare June 19, 2024 11:34
@kkourt kkourt self-requested a review June 20, 2024 08:55
Copy link
Contributor

@kkourt kkourt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@mtardy mtardy merged commit 2e9e701 into v1.0 Jul 1, 2024
29 of 30 checks passed
@mtardy mtardy deleted the pr/mtardy/backport-v1.0-stacktrace-memory branch July 1, 2024 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/backport This PR provides functionality previously merged into master. release-note/bug This PR fixes an issue in a previous release of Tetragon.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants