pageserver: make local timeline deletion infallible

[jcsp]

Problem

In #9589, timeline offload code is modified to return an explicit error type rather than propagating anyhow::Error. One of the 'Other' cases there is I/O errors from local timeline deletion, which shouldn't need to exist, because our policy is not to try and continue running if the local disk gives us errors.

Summary of changes

• Make delete_local_timeline_directory and use .fatal_err( on I/O errors

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[jcsp]

@erikgrinaker for context, our internal note on infallible local I/O is https://docs.neon.build/storage/handling_io_and_logical_errors.html (not all code respects this yet)

[github-actions]

5354 tests run: 5129 passed, 3 failed, 222 skipped (full report)

---

Failures on Postgres 17

• test_sharding_split_failures[failure0]: debug-x86-64
• test_sharding_split_failures[failure12]: debug-x86-64
• test_sharding_split_failures[failure9]: debug-x86-64

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_sharding_split_failures[debug-pg17-failure0] or test_sharding_split_failures[debug-pg17-failure12] or test_sharding_split_failures[debug-pg17-failure9]"

Flaky tests (2)

Postgres 17

• test_sharding_split_smoke: debug-x86-64
• test_scrubber_scan_pageserver_metadata[4]: debug-x86-64

Code coverage* (full report)

• functions: 31.5% (7772 of 24690 functions)
• lines: 48.9% (61030 of 124690 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
4246622 at 2024-11-01T18:38:57.841Z :recycle:}

[erikgrinaker]

for context, our internal note on infallible local I/O is https://docs.neon.build/storage/handling_io_and_logical_errors.html (not all code respects this yet)

Makes sense, thanks for the background.