Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make master outlive confdata out of shared memory errors #946

Merged
merged 21 commits into from
Dec 9, 2023

Conversation

DrDet
Copy link
Contributor

@DrDet DrDet commented Dec 3, 2023

Here're two new approaches:

Events throttling

To prevent OOM errors on confdata overflows we start to throttle events, when too little shared memory left. There're two modes:

  • Soft OOM (85%) - replay only existing keys related events until server restart. Ignore new keys.
  • Hard OOM (95%) - replay nothing. Ignore all events, don't read binlog at all.

If server enters any OOM modes, it can't go back out of it. Manual server restart is needed for that.

External preventive allocation heuristic checks

Before any action, that internally invokes allocations, we explicitly check that allocator has enough memory for that.
To achieve this we need to somehow calculate upper-bound for memory that will be required by the action in each particular case.
There're two 'kinds' of allocations:

  1. Allocation made by our runtime (array<mixed>::mutate_if_shared(), etc..). It's not too complicated, because it's our own code and we can calculate it easily.
  2. Allocation made by STL, out of std::map<> which uses our shared memory allocator. To handle this we will use common sense, some knowledge about search trees internal structure and compile time upper-bound for tree node size from our allocator:
    static constexpr size_t max_value_type_size() {

New stats for confdata

This PR also includes writing confdata metrics to statshouse:

  • kphp_confdata_memory
  • kphp_confdata_events
  • kphp_confdata_sections_by_count
  • kphp_confdata_update_fails

New option for blacklisting confdata events

--confdata-force-ignore-keys-prefix — an emergency option, e.g. 'highload.vid*', to forcibly drop keys from snapshot/binlog; may be used multiple times.

Timeouts for reading binlog

To prevent master hanging in master_cron(), a timeout for reading a next binlog chunk is added, controlled by --confdata-update-timeout. Every second (every cron iteration) we allow binlog reader to consume next event no more than timeout seconds. That allows master to perform other work.

Note, that an option for reading binlog in a separate thread can't be implemented, since the master process is also fundamentally single-threaded (lots of globals, global allocator, global allocator switches between heap/shared memory, etc.).

@DrDet DrDet added enhancement New feature or request runtime Feature related to runtime labels Dec 3, 2023
@DrDet DrDet added this to the next milestone Dec 3, 2023
@DrDet DrDet requested a review from tolk-vm December 3, 2023 19:08
@DrDet DrDet self-assigned this Dec 3, 2023
@DrDet DrDet force-pushed the dvaksman/make-master-outlive-confdata-overflow branch 4 times, most recently from 22120c5 to a263915 Compare December 9, 2023 13:41
server/confdata-binlog-replay.cpp Outdated Show resolved Hide resolved
server/confdata-binlog-replay.cpp Show resolved Hide resolved
server/confdata-binlog-replay.cpp Outdated Show resolved Hide resolved
server/confdata-binlog-replay.cpp Outdated Show resolved Hide resolved
@DrDet DrDet force-pushed the dvaksman/make-master-outlive-confdata-overflow branch from 3db0400 to 7d80e52 Compare December 9, 2023 18:38
server/confdata-stats.h Outdated Show resolved Hide resolved
@DrDet DrDet force-pushed the dvaksman/make-master-outlive-confdata-overflow branch from 7d80e52 to b55eb00 Compare December 9, 2023 18:43
tolk-vm
tolk-vm previously approved these changes Dec 9, 2023
@DrDet DrDet force-pushed the dvaksman/make-master-outlive-confdata-overflow branch from d4b4515 to 8e97de6 Compare December 9, 2023 19:25
@DrDet DrDet merged commit 34f5177 into master Dec 9, 2023
5 checks passed
@DrDet DrDet deleted the dvaksman/make-master-outlive-confdata-overflow branch December 9, 2023 20:11
DedAzaMarks pushed a commit to DedAzaMarks/kphp that referenced this pull request Dec 13, 2023
- events throttling approach with soft/hard OOM mode
- external allocation heuristic checks to prevent OOM
- timeout for reading binlog on updates to prevent master freezing 
- new emergency option for blacklisting confdata events
- extend confdata monitoring in StatsHouse and Graphana 

Co-authored-by: Aleksandr Kirsanov <unserialize.alias@gmail.com>
mt-omarov pushed a commit to mt-omarov/kphp that referenced this pull request Feb 26, 2024
- events throttling approach with soft/hard OOM mode
- external allocation heuristic checks to prevent OOM
- timeout for reading binlog on updates to prevent master freezing 
- new emergency option for blacklisting confdata events
- extend confdata monitoring in StatsHouse and Graphana 

Co-authored-by: Aleksandr Kirsanov <unserialize.alias@gmail.com>
mt-omarov pushed a commit to mt-omarov/kphp that referenced this pull request Feb 26, 2024
- events throttling approach with soft/hard OOM mode
- external allocation heuristic checks to prevent OOM
- timeout for reading binlog on updates to prevent master freezing 
- new emergency option for blacklisting confdata events
- extend confdata monitoring in StatsHouse and Graphana 

Co-authored-by: Aleksandr Kirsanov <unserialize.alias@gmail.com>
mt-omarov pushed a commit to mt-omarov/kphp that referenced this pull request Feb 26, 2024
- events throttling approach with soft/hard OOM mode
- external allocation heuristic checks to prevent OOM
- timeout for reading binlog on updates to prevent master freezing 
- new emergency option for blacklisting confdata events
- extend confdata monitoring in StatsHouse and Graphana 

Co-authored-by: Aleksandr Kirsanov <unserialize.alias@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request runtime Feature related to runtime
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants