Epic: ingestion performance phase 2

[jcsp]

This epic tracks work planned for the start of 2025.

It follows on from #9329 and other related work done recently.

The specific tasks are subject to change based on feedback from benchmarking: the high level goal is consolidate our gains from recent improvements in the regime of ~100MB/s ingest.

Tasks

Preview Give feedback

1. pageserver: improve L0 compaction performance #10694
   12 of 14
   a/performance c/storage/pageserver +2
   
2. Investigate WAL ingest performance for bulk writes #9789
   a/performance c/storage +2
   
3. pageserver: backpressure when delta layer count is high #5415
   c/storage/pageserver t/feature +2
   
4. storcon: reconciles can starve optimizations, which can starve auto-splits #10762
   a/performance a/scalability c/storage/controller +3
   
5. pageserver: upload/delete race #10283
   c/storage/pageserver t/bug +2
   
6. safekeeper: optimize WAL decoding #10097
   a/performance c/storage/safekeeper +2
   
7. Benchmark creating many tables and databases #9986
   a/benchmark c/storage +2
   
8. safekeeper: increase segment size #9687
   a/performance c/storage/safekeeper +2
   
9. Epic: revamp pageserver backpressure #8390
   3 of 5
   a/performance c/storage/pageserver t/bug triaged +4