Epic: optimize WAL ingest pipeline

[erikgrinaker]

Motivation

We should investigate WAL ingestion and optimize for end-to-end throughput. In particular for migrations of new users, typically pg_restore or logical replication.

DoD

Ingestion should saturate hardware (either disk or network IO) on at least one of compute, safekeeper, or pageserver.

Implementation ideas

TBD

Tasks

Tasks

Preview Give feedback

1. Investigate WAL ingest performance for logical messages #9642
   a/performance c/storage +2
   
2. Investigate WAL ingest performance for bulk writes #9789
   a/performance c/storage +2
   
3. Investigatepg_restore performance #9623
   a/benchmark a/performance c/storage +3
   
4. pageserver: add L0 compute backpressure (replace flush backpressure) #10095
   a/performance c/storage/pageserver +2
   
5. pageserver: improve flush upload queue parallelism #10096
   a/performance c/storage/pageserver +2
   
6. safekeeper: optimize WAL decoding #10097
   a/performance c/storage/safekeeper +2
   
7. remote_storage: revisit throttling/ratelimiting #3698
   c/storage/pageserver +1
8. remote_storage: add request/response size histograms #9819
   a/observability c/storage +2
9. remote_storage: increase concurrency_limit to 200 #10038
   a/performance c/storage/pageserver +2
   
10. pageserver: slow get_rel_exists() during WAL ingestion with many relations #9855
    a/performance c/storage/pageserver +2
    
11. safekeeper: optimize writing of zeroed segments #9662
    a/performance c/storage/safekeeper +2
    
12. safekeeper: avoid or optimize segment fsyncs #9664
    a/performance c/storage/safekeeper +2
    
13. safekeeper: optimize control file fsyncs on commit_lsn advancement #9663
    a/performance c/storage/safekeeper +2
    
14. safekeeper: eagerly return AppendResponse with flush_lsn on segment fsync #9688
    a/performance c/storage/safekeeper +2
    
15. CPU/heap profiling at runtime #9534
    a/observability c/storage +2
    
16. safekeeper: increase segment size #9687
    a/performance c/storage/safekeeper +2
    
17. safekeeper: use tokio-epoll-uring for file IO #9741
    a/performance c/storage/safekeeper +2
18. safekeeper: improve batching of pipelined AppendRequest #9689
    a/performance c/storage/safekeeper +2
19. safekeeper: return AppendResponse on txn commit #9690
    a/performance c/storage/safekeeper +2
    

Other related tasks and epics

• Investigate bulk load performance #1263

[erikgrinaker]

Closing in favor of #10160.