Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault / memory leak #76

Closed
shreevatsa opened this issue Sep 12, 2021 · 5 comments
Closed

Segmentation fault / memory leak #76

shreevatsa opened this issue Sep 12, 2021 · 5 comments

Comments

@shreevatsa
Copy link

Hi, thank you for this really nice crate. I couldn't get most other profilers to work on a macOS laptop (x86_64-apple-darwin), while this one works easily and generates useful flamegraphs.

Unfortunately it seems to be causing a segmentation fault in my program when I use it. There's no backtrace and the location of the crash seems to be non-deterministic so I have only indirect evidence:

  • There was a segmentation fault when I was using pprof::ProfilerGuard and it went away when I removed it.
  • Running the program under leak sanitizer showed a leak from objects allocated from here I think (see output in comment elsewhere), though maybe I've misunderstood.

Maybe the issue is not with this pprof directly but with the combination of this and some other dependency, but I doubt I have the ability to debug further. Just filing this here in case the issue is obvious or it helps somehow,

@YangKeao
Copy link
Member

There was a segmentation fault when I was using pprof::ProfilerGuard and it went away when I removed it.

There are some known situations where pprof-rs may cause segmentation fault:

If your program is getting backtrace (e.g. other profilers are running, the Error generates a backtrace automatically...) while the profiling signal arrives, the program will panic. It has been mentioned in README.

Running the program under leak sanitizer showed a leak from objects allocated from here I think (see output in comment elsewhere), though maybe I've misunderstood.

I don't know how the LeakSanitizer handles the lazy_static objects. It seems that all memory allocated from Profiler::new is regarded as a leak. However, all these memory is referenced from the global lazy_static variable 🤦‍♂️ .

@shreevatsa
Copy link
Author

There are some known situations where pprof-rs may cause segmentation fault:

If your program is getting backtrace (e.g. other profilers are running, the Error generates a backtrace automatically...)

Ah, it was the latter — this was my first Rust program more-or-less, and initially I had .unwrap() everywhere, and at some point I gathered the (incorrect it looks like?) impression that it was "better" to use Result types and change every .unwrap() to ? — but this resulted in no longer getting stack traces on error, so I added it back as here. So I guess that's the problem here, trying to do both profiling and have Error generate backtraces. (In the next program I guess I'll just use unwrap() except where I plan to handle the error… not yet sure how to get stack traces in that case though; losing stack traces seems like a reason to avoid Error/Result types!)

Running the program under leak sanitizer showed a leak from objects allocated from here I think (see output in comment elsewhere), though maybe I've misunderstood.

I don't know how the LeakSanitizer handles the lazy_static objects. It seems that all memory allocated from Profiler::new is regarded as a leak. However, all these memory is referenced from the global lazy_static variable 🤦‍♂️ .

Thanks, I was suspecting that there wasn't an actual leak. Looks like LeakSanitizer blamed pprof-rs for incorrect reasons, but removing it made the segmentation fault go away.

@YangKeao
Copy link
Member

YangKeao commented Nov 2, 2021

@shreevatsa I'm sorry. There are memory leaks 😿 and they have been fixed by the latest version.

If you are still facing segmentation fault, you could try let guard = pprof::ProfilerGuardBuilder::default().frequency(1000).blocklist(&["libc", "libgcc", "pthread"]).build().unwrap(); . It will resolve the deadlock in dl_iterate_phdr (but will ignore the sample in libc, libgcc and pthread.

@YangKeao
Copy link
Member

As #84 has merged, I would close this issue. If you have more questions / bugs, feel free to reopen or create a new issue 😃

@shreevatsa
Copy link
Author

@YangKeao Thank you! Sorry I couldn't get around to testing again after the fix, but I will re-enable profiling in my toy project, and let you know if any issue happens again. Thanks for the fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants