safekeeper: avoid or optimize segment fsyncs

[erikgrinaker]

The WAL receiver attempts to batch writes and avoid fsyncs via NoFlushAppendRequest, only flushing every second:

neon/safekeeper/src/receive_wal.rs

Lines 537 to 543 in 123816e

	// Don't flush the WAL on every append, only periodically via flush_ticker.
	// This batches multiple appends per fsync. If the channel is empty after
	// sending the reply, we'll schedule an immediate flush.
	if let ProposerAcceptorMessage::AppendRequest(append_request) = msg {
	msg = ProposerAcceptorMessage::NoFlushAppendRequest(append_request);
	dirty = true;
	}

However, we unconditionally fsync after writing out a complete 16 MB segment:

neon/safekeeper/src/wal_storage.rs

Lines 297 to 298 in b54b632

	// If we reached the end of a WAL segment, flush and close it.
	self.fdatasync_file(&file).await?;

We also unconditionally fsync 3 times when we create a new segment:

neon/safekeeper/src/wal_storage.rs

Line 270 in 4282061

if let Err(e) = durable_rename(&tmp_path, &wal_file_partial_path, !self.no_sync).await {

This implies that we'll fsync four times after every 16 MB. If we want to hit 1 GB/s, that's 256 fsyncs. We should try to optimize this.

Also note that the control file is fsynced every time the commit LSN crosses a segment boundary. That's an additional 64 fsyncs per second. See #9663.

[erikgrinaker]

durable_rename() fsyncs (3 in total) are responsible for 50% of WAL ingest duration with large files. We should try to do these fsyncs concurrently, and/or defer them until the first WAL segment flush and piggyback on it.

[erikgrinaker]

We removed an unnecessary fsync in for every segment in #9686, and moved control file flushes off of the WAL ingest path in #9698.

If we need to further reduce fsyncs, that's probably easiest done by increasing the segment size from 16 MB to e.g. 128 MB, see #9687.