Load generation App Updates #1509

PaurushGarg · 2023-11-28T07:53:35Z

Description:

Improves OTLP Metric and Traces Load Generation

Remove configurable observation interval, flush interval, and datapoints from metrics- as they are unused and not-required in the context of load-generation required for the performance.
Corrects the command line parameters of metric generation - as per the way it is used from load-gen.
updates the test-framework to use the latest load-gen app.
correctly sets up metric and trace provider to produce the right load (metric/traces).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Danielzolty · 2023-11-30T01:06:37Z

Here are performance results for 100, 1,000, and 5,000 TPS for metric tests:

100:
{"testcase":"otlp_metric_mock","instanceType":"m5.2xlarge","receivers":["otlp"],"processors":["batch"],"exporters":["logging","awsemf"],"dataType":"otlp","dataMode":"metric","dataRate":100,"avgCpu":0.350000239262089,"avgMem":109.9917312,"maxCpu":0.5000363635444206,"maxMem":111.120384,"commitId":"dummy_commit","collectionPeriod":10,"testingAmi":"soaking_linux"}

1,000:
{"testcase":"otlp_metric_mock","instanceType":"m5.2xlarge","receivers":["otlp"],"processors":["batch"],"exporters":["logging","awsemf"],"dataType":"otlp","dataMode":"metric","dataRate":1000,"avgCpu":2.7616599662192955,"avgMem":141.03784106666666,"maxCpu":8.99966334149369,"maxMem":154.710016,"commitId":"dummy_commit","collectionPeriod":10,"testingAmi":"soaking_linux"}

5,000:
{"testcase":"otlp_metric_mock","instanceType":"m5.2xlarge","receivers":["otlp"],"processors":["batch"],"exporters":["logging","awsemf"],"dataType":"otlp","dataMode":"metric","dataRate":5000,"avgCpu":14.974639151005709,"avgMem":224.80576511999996,"maxCpu":21.39946276862715,"maxMem":254.705664,"commitId":"dummy_commit","collectionPeriod":10,"testingAmi":"soaking_linux"}

vasireddy99

Did 1st pass on this PR

vasireddy99 · 2023-12-04T20:03:39Z

load-generator/README.md

@@ -49,7 +41,7 @@ The following is a list of additional optional command line arguments applicable

 ### StatsD Metrics Load Test Sample Command,
 ```
-./gradlew :load-generator:run --args="metric --metricCount=100 --datapointCount=10 --observationInterval=1000 -u=localhost:8125 -d=statsd"


observationInterval=1000 should we have observation interval configurable.

I think as sample app, providing configurable observationInterval is good, but in load generator context and how it is used through test-framework; I think it would be unused functionality and could create problem as well.

rapphil · 2023-12-05T21:52:51Z

...nerator/src/main/java/com/amazon/opentelemetry/load/generator/emitter/OtlpMetricEmitter.java

-      log.error("Metric provider was not found!");
+
+    private void createCounters(Meter meter) {
+        if (meter != null) {


nit: this could be replaced by an early return.

rapphil · 2023-12-05T21:53:12Z

...nerator/src/main/java/com/amazon/opentelemetry/load/generator/emitter/OtlpMetricEmitter.java

+                                }
+                        );
+            }
+        } else {


rapphil · 2023-12-05T22:02:12Z

load-generator/src/main/java/com/amazon/opentelemetry/load/generator/emitter/MetricEmitter.java

-    log.info("Generating metrics at a rate of(ms) : {}" , this.param.getObservationInterval());
-    scheduler.scheduleAtFixedRate(emitter, 0, this.param.getObservationInterval(), TimeUnit.MILLISECONDS);
+    log.info("Generating metrics at a rate of(ms) : {}" , FLUSH_INTERVAL);
+    scheduler.scheduleAtFixedRate(emitter, 0, FLUSH_INTERVAL, TimeUnit.MILLISECONDS);


this simplifies things a bit. Should we add an warning/raise an exception in case the emitter function is not able to complete in less than 1s? Otherwise the ingestion rate will not be correct right?

I am unsure how to implement this. scheduleAtFixedRate only throws these three exceptions:

RejectedExecutionException – if the task cannot be scheduled for execution
NullPointerException – if command or unit is null
IllegalArgumentException – if period less than or equal to zero

rapphil · 2023-12-05T22:05:00Z

load-generator/src/main/java/com/amazon/opentelemetry/load/generator/emitter/TraceEmitter.java

@@ -24,7 +24,7 @@ public abstract class TraceEmitter implements Emitter {

  @Override
  public void start(Runnable emitter) {
-    scheduler.scheduleAtFixedRate(emitter, TimeUnit.SECONDS.toMillis(5000),
+    scheduler.scheduleAtFixedRate(emitter, 0,
        TimeUnit.SECONDS.toNanos(1) / this.param.getRate(), TimeUnit.NANOSECONDS);


I think you need to fix this as well no? this needs to be scheduled every second and emitter should emitt the rate number of spans?

Raphael, in my opinion, this is already fixed. works little differently than metrics. nextdatapoint is called no of rate times, which produces that many traces in a second, which gets batched and exported at 1s frequency.

bryan-aguilar · 2023-12-06T19:55:20Z

Improves OTLP Metric Load Generation

How? By doing what? What changes are being made in this PR?

bryan-aguilar · 2023-12-06T19:57:03Z

load-generator/src/main/java/com/amazon/opentelemetry/load/generator/emitter/Emitter.java

@@ -21,6 +21,9 @@
 public interface Emitter {

  int NUM_THREADS = 5;
+
+  final long FLUSH_INTERVAL = 1000;


Why is this no longer configurable?

I think, for load gen its not required to be configured, as we have TPS (100, 1000, 5000) batched at 1s. Adding flush_interval just adds unused complexity.
#1509 (comment)

bryan-aguilar · 2023-12-06T20:00:57Z

terraform/testcases/prometheus_mock/otconfig.tpl

-        scrape_interval: 15s
+        scrape_interval: 1s


Doesn't changing the scrape interval affect performance? Why is this change necessary?

yes it should affect performance by factor of 15. This is to match the metric load as mentioned in the performance report.

Reverted. Will pull into separate PR as discussed.

PaurushGarg · 2023-12-15T19:15:47Z

Improves OTLP Metric Load Generation

How? By doing what? What changes are being made in this PR?

Updated PR body.

PaurushGarg added 2 commits November 15, 2023 08:44

Adding aws_sdk_2 trace prefix

782cd11

Adding draft changes for otlp counter metric

0df5553

PaurushGarg requested a review from a team as a code owner November 28, 2023 07:53

PaurushGarg requested a review from Danielzolty November 28, 2023 07:53

PaurushGarg added 6 commits November 28, 2023 09:53

Removing build and format error

244335b

Removing debug changes from PR

10ad682

Revising the rate to 1s

2d5df9e

Revising statsd aggregation interval

f3d08a9

Changing Trace Emitter OTLP Queue Size and Export time

31d252a

Reverting inadvertenet statsd changes

988bbeb

PaurushGarg added 2 commits December 1, 2023 10:19

Reverting some unintended changes

8a22b53

Refactoring

66066f9

PaurushGarg requested review from rapphil, vasireddy99, bryan-aguilar and humivo December 1, 2023 18:37

Use latest load-gen image

49308ca

vasireddy99 reviewed Dec 4, 2023

View reviewed changes

rapphil reviewed Dec 5, 2023

View reviewed changes

bryan-aguilar reviewed Dec 6, 2023

View reviewed changes

PaurushGarg added 2 commits December 11, 2023 11:20

Resolving comments - early return

b9cadba

Reverting changes to prometheus config

393c9b2

PaurushGarg changed the title ~~Load generation update metrics~~ Load generation App Updates Dec 15, 2023

PaurushGarg requested review from rapphil, bryan-aguilar and vasireddy99 December 15, 2023 19:15

bryan-aguilar approved these changes Dec 20, 2023

View reviewed changes

bryan-aguilar merged commit a371a93 into terraform Dec 20, 2023
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load generation App Updates #1509

Load generation App Updates #1509

PaurushGarg commented Nov 28, 2023 •

edited

Loading

Danielzolty commented Nov 30, 2023

vasireddy99 left a comment

vasireddy99 Dec 4, 2023

PaurushGarg Dec 7, 2023

rapphil Dec 5, 2023

PaurushGarg Dec 11, 2023

rapphil Dec 5, 2023

PaurushGarg Dec 11, 2023

rapphil Dec 5, 2023

PaurushGarg Dec 15, 2023

rapphil Dec 5, 2023

PaurushGarg Dec 11, 2023

bryan-aguilar commented Dec 6, 2023

bryan-aguilar Dec 6, 2023

PaurushGarg Dec 11, 2023

bryan-aguilar Dec 6, 2023

PaurushGarg Dec 11, 2023

PaurushGarg Dec 15, 2023

PaurushGarg commented Dec 15, 2023

Load generation App Updates #1509

Load generation App Updates #1509

Conversation

PaurushGarg commented Nov 28, 2023 • edited Loading

Danielzolty commented Nov 30, 2023

vasireddy99 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryan-aguilar commented Dec 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PaurushGarg commented Dec 15, 2023

PaurushGarg commented Nov 28, 2023 •

edited

Loading