Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PROF-9470] Align heap recorder cleanup with GC activity (second try) #4020

Merged
merged 33 commits into from
Oct 24, 2024
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
8a38d67
[PROF-9470] Align heap recorder cleanup with GC activity
AlexJF Sep 6, 2024
c5d6bcf
[PROF-9470] Continue skipping updates of age == 0
AlexJF Sep 13, 2024
a789481
[PROF-9470] Fixing issues
AlexJF Sep 16, 2024
1d44add
[PROF-9470] Improvements
AlexJF Sep 30, 2024
4226d26
Fix inline position
AlexJF Sep 30, 2024
639b856
More improvements
AlexJF Sep 30, 2024
d87b7ed
More debugging to help with failing test troubleshooting
AlexJF Sep 30, 2024
f0df1a0
Remove inspection since it turns out not to be that safe
AlexJF Oct 1, 2024
5efbab5
Ensure heap recorder is in a deterministic state on spec
AlexJF Oct 1, 2024
0ef68dd
Re-add commented-out inspection
AlexJF Oct 1, 2024
abf7166
Minor: Remove unused import
ivoanjo Oct 14, 2024
67ae0c8
Minor: Tweak text in comment
ivoanjo Oct 15, 2024
dc900a9
Remove `pending_update` mechanism from Heap recorder
ivoanjo Oct 15, 2024
b2cc184
Minor: Remove unneeded `#include`
ivoanjo Oct 17, 2024
b83ac3a
Continue refactoring heap cleanup after GC to be best-effort
ivoanjo Oct 17, 2024
e90ee7d
Remove `last_update_major_gc_since` as it was not working
ivoanjo Oct 17, 2024
585d6e4
Simplify `heap_recorder_update` logic now that `full_update` decision…
ivoanjo Oct 17, 2024
6f650d6
Hide `heap_recorder_update` from heap recorder API
ivoanjo Oct 17, 2024
ba35aed
Minor: Tweak exception error message
ivoanjo Oct 17, 2024
0e97c6c
Optimization: Skip updating object sizes when not doing a full update
ivoanjo Oct 17, 2024
adb210d
Add TODO on metrics to discuss with @alexjf
ivoanjo Oct 17, 2024
f581076
Fix segfaults when calling `_native_debug_heap_recorder` in tests
ivoanjo Oct 17, 2024
395da1b
Add integration-style spec for heap cleanup after GC
ivoanjo Oct 22, 2024
0343ba4
Add unit test for heap cleanup after GC only testing young objects
ivoanjo Oct 22, 2024
7b21386
Introduce `StackRecorder.for_testing` helper
ivoanjo Oct 22, 2024
d5368c5
Convert `StackRecorder._native_initialize` to use kw args
ivoanjo Oct 23, 2024
770b1aa
Introduce setting to control heap clean after GC
ivoanjo Oct 23, 2024
ab1832a
Remove "only-profiling-alloc-ddprof" testing configuration
ivoanjo Oct 23, 2024
ce1fb24
Add benchmarking variant for heap with clean after GC
ivoanjo Oct 23, 2024
8aadb81
Add test coverage for not updating heap half-way during serialization
ivoanjo Oct 23, 2024
4c659a5
Add test coverage for minimum period between heap updates
ivoanjo Oct 23, 2024
6494eee
Minor: Align `heap_clean_after_gc_enabled` native default value with …
ivoanjo Oct 23, 2024
dd653ce
Minor: Split heap recorder stats into young/non-young
ivoanjo Oct 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .gitlab/benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,22 +95,23 @@ only-profiling-alloc:
DD_PROFILING_ALLOCATION_ENABLED: "true"
ADD_TO_GEMFILE: "gem 'datadog', github: 'datadog/dd-trace-rb', ref: '$CI_COMMIT_SHA'"

only-profiling-alloc-ddprof:
only-profiling-heap:
extends: .benchmarks
variables:
DD_BENCHMARKS_DDPROF: "true"
DD_BENCHMARKS_CONFIGURATION: only-profiling
DD_PROFILING_ENABLED: "true"
DD_PROFILING_ALLOCATION_ENABLED: "true"
DD_PROFILING_EXPERIMENTAL_HEAP_ENABLED: "true"
ADD_TO_GEMFILE: "gem 'datadog', github: 'datadog/dd-trace-rb', ref: '$CI_COMMIT_SHA'"

only-profiling-heap:
only-profiling-heap-clean-after-gc:
extends: .benchmarks
variables:
DD_BENCHMARKS_CONFIGURATION: only-profiling
DD_PROFILING_ENABLED: "true"
DD_PROFILING_ALLOCATION_ENABLED: "true"
DD_PROFILING_EXPERIMENTAL_HEAP_ENABLED: "true"
DD_PROFILING_HEAP_CLEAN_AFTER_GC_ENABLED: "true"
ADD_TO_GEMFILE: "gem 'datadog', github: 'datadog/dd-trace-rb', ref: '$CI_COMMIT_SHA'"

profiling-and-tracing:
Expand Down
9 changes: 1 addition & 8 deletions benchmarks/profiler_gc.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,7 @@

class ProfilerGcBenchmark
def create_profiler
@recorder = Datadog::Profiling::StackRecorder.new(
cpu_time_enabled: true,
alloc_samples_enabled: false,
heap_samples_enabled: false,
heap_size_enabled: false,
heap_sample_every: 1,
timeline_enabled: true,
)
@recorder = Datadog::Profiling::StackRecorder.for_testing(timeline_enabled: true)
@collector = Datadog::Profiling::Collectors::ThreadContext.for_testing(recorder: @recorder, timeline_enabled: true)

# We take a dummy sample so that the context for the main thread is created, as otherwise the GC profiling methods do
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_memory_sample_serialize.rb
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ def setup
@retain_every = (ENV['RETAIN_EVERY'] || '10').to_i
@skip_end_gc = ENV['SKIP_END_GC'] == 'true'
@recorder_factory = proc {
Datadog::Profiling::StackRecorder.new(
Datadog::Profiling::StackRecorder.for_testing(
cpu_time_enabled: false,
alloc_samples_enabled: true,
heap_samples_enabled: @heap_samples_enabled,
Expand Down
9 changes: 1 addition & 8 deletions benchmarks/profiler_sample_gvl.rb
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,7 @@ def initialize
end

def create_profiler
@recorder = Datadog::Profiling::StackRecorder.new(
cpu_time_enabled: true,
alloc_samples_enabled: false,
heap_samples_enabled: false,
heap_size_enabled: false,
heap_sample_every: 1,
timeline_enabled: true,
)
@recorder = Datadog::Profiling::StackRecorder.for_testing(timeline_enabled: true)
@collector = Datadog::Profiling::Collectors::ThreadContext.for_testing(
recorder: @recorder,
waiting_for_gvl_threshold_ns: 0,
Expand Down
9 changes: 1 addition & 8 deletions benchmarks/profiler_sample_loop_v2.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,7 @@ class ProfilerSampleLoopBenchmark
PROFILER_OVERHEAD_STACK_THREAD = Thread.new { sleep }

def create_profiler
@recorder = Datadog::Profiling::StackRecorder.new(
cpu_time_enabled: true,
alloc_samples_enabled: false,
heap_samples_enabled: false,
heap_size_enabled: false,
heap_sample_every: 1,
timeline_enabled: false,
)
@recorder = Datadog::Profiling::StackRecorder.for_testing
@collector = Datadog::Profiling::Collectors::ThreadContext.for_testing(recorder: @recorder)
end

Expand Down
9 changes: 1 addition & 8 deletions benchmarks/profiler_sample_serialize.rb
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,7 @@ class ProfilerSampleSerializeBenchmark

def create_profiler
timeline_enabled = ENV['TIMELINE'] == 'true'
@recorder = Datadog::Profiling::StackRecorder.new(
cpu_time_enabled: true,
alloc_samples_enabled: false,
heap_samples_enabled: false,
heap_size_enabled: false,
heap_sample_every: 1,
timeline_enabled: timeline_enabled,
)
@recorder = Datadog::Profiling::StackRecorder.for_testing(timeline_enabled: timeline_enabled)
@collector = Datadog::Profiling::Collectors::ThreadContext.for_testing(recorder: @recorder, timeline_enabled: timeline_enabled)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -815,6 +815,9 @@ VALUE thread_context_collector_sample_after_gc(VALUE self_instance) {

state->stats.gc_samples++;

// Let recorder do any cleanup/updates it requires after a GC step.
recorder_after_gc_step(state->recorder_instance);

// Return a VALUE to make it easier to call this function from Ruby APIs that expect a return value (such as rb_rescue2)
return Qnil;
}
Expand Down Expand Up @@ -1441,7 +1444,8 @@ void thread_context_collector_sample_allocation(VALUE self_instance, unsigned in
class_name = ruby_value_type_to_class_name(type);
}
} else {
// Fallback for objects with no class
// Fallback for objects with no class. Objects with no class are a way for the Ruby VM to mark them
// as internal objects; see rb_objspace_internal_object_p for details.
class_name = ruby_value_type_to_class_name(type);
}
} else if (type == RUBY_T_IMEMO) {
Expand Down
Loading
Loading