[BUG] composite_terms-keyword has increased latency with Lucene 10 index

[expani]

Describe the bug

The latency for composite_terms-keyword operation in Big5 has increased by 10-15% with the upgrade to Lucene 10. When using the index from OS 2.19 but searching using OS 3.0 server the latency is better. So, we have narrowed down the cause to something wrong with Index format or we misconfiguring/not configuring something with the Lucene 10 upgrade.

OS 3.0 * OS 2.19 indicates that the OpenSearch server was running 3.0 whereas the index used was created in OS 2.19. This was done to eliminate any suspicion of the bug arising from an indexing change in Lucene.

Metric Name	Operation/Query name	OS 2.19	OS 3.0	OS 3.0 * OS 2.19	Unit
50th percentile latency	composite_terms-keyword	385.519	437.748	363.538	ms
90th percentile latency	composite_terms-keyword	393.628	441.061	372.952	ms
99th percentile latency	composite_terms-keyword	425.658	451.375	382.752	ms
100th percentile latency	composite_terms-keyword	447.978	457.927	383.355	ms
50th percentile service time	composite_terms-keyword	384.585	436.741	362.469	ms
90th percentile service time	composite_terms-keyword	392.626	440.274	372.171	ms
99th percentile service time	composite_terms-keyword	424.494	450.533	381.558	ms
100th percentile service time	composite_terms-keyword	447.026	456.523	382.67	ms

Related component

Search:Performance

To Reproduce

Run composite_terms-keyword with Big5 workload and compare with OS 2.19

Expected behavior

composite_terms-keyword should perform the same with OS 3.0 index as it's doing with OS 2.19 index

Additional Details

Meta