Add feedback mechanism to predict max throughput of the OpenSearch cluster

[rishabh6788]

Is your feature request related to a problem? Please describe

Currently, opensearch-benchmark requires users to manually specify the number of clients and target throughput when running performance benchmarks. This requires multiple trial-and-error runs to find the optimal load that a system can handle while maintaining acceptable performance. We need an automated way to discover the maximum sustainable load while keeping response latency and other performance parameters within acceptable limits.

Describe the solution you'd like

Add capability for opensearch-benchmark to automatically determine the maximum number of parallel clients a system can handle while maintaining latency or any other performance parameter below a user-defined threshold. Instead of specifying client count and throughput, users will specify:

• Total benchmark duration
• Maximum acceptable query latency threshold

The tool will:

1. Start with minimal load.
2. Gradually ramp up the number of clients.
3. Monitor query latency or any other defined performance parameter.
4. Stop adding clients when threshold breaches threshold.
5. Report maximum sustainable client count along with standard metrics.

Describe alternatives you've considered

No response

Additional context

No response

Acceptance Criteria

• Command line interface supports duration and latency threshold parameters
• Implements gradual client scaling with latency monitoring
• Stops scaling when latency threshold is exceeded
• Generates enhanced report with scaling metrics
• Includes safety measures and proper error handling
• Documentation updated to reflect new functionality