Skip to content

Latest commit

 

History

History
566 lines (516 loc) · 11.9 KB

LoadTesting.md

File metadata and controls

566 lines (516 loc) · 11.9 KB

Mapping Performance - JUnit Perf Load Testing

Challenge

  • The general performance and behavior of the mapping engine under high payload are unknown.

Idea

  • Implementation of load tests to measure the performance according to different mapping scenarios.
  • Generation of statistical reports about key performance metrics as e.g average execution time or min, max latency.

Solution

  • Implementation of benchmark/performance test scenarios using JUnitPerf annotations in usage of existing mapping engine test functions.
  • Integration of load tests in the Travis build process to provide direct feedback through an exported html report, accessible via web-browser.

Performance Testing

Environment:

Processor: Intel(R) Core(TM) i7-8650U CPU @ 1.9GHz 2.11 Ghz

Installed memory (RAM): 32GB

System: 64-bit Oberating System, x64-based processor, Windows 10 Enterprise

Configuration Parameters:

The selected JUnitPerf framework allows to annotate test functions with custom java annotations to run the selected tests multiple times while gathering statistical information. The annotation syntax allows the configuration of the following parameters.

threads - The total number of threads to use during test execution

warmUpMs - The warm up period in ms, test logic will be executed during warm up, but results will not be considered during statistics evaluation

durationMs - The total time to run the test in millisecs (ms) (includes warmup period)

maxExecutionsPerSecond - The maximum number of iteration per second

rampUpPeriodMs - The framework ramps up its executions per second smoothly over the duration of this period

Scenario 1 - Binary Data Mapping

For the reason that the normalization of binary data into a mapped output requires conversion, the Binary Data Mapping tests described in the following only imply cases that make use of a converter.

With Built-In Converter:

  • Test-Case 1: 1 Built-in (Java) Converter Function

With JavaScript Converter:

  • Test-Case 2: Two binary data points, mapping with Double Nested Converter Functions (Built-in (Java) + Javascript)
  • Test-Case 3: Triple Nested Converter Functions (Built-in + Built-in + Javascript)

Each of the tests is executed 4 times with different configuration parameters:

Execution threads warmUpMs durationMs maxExecutionsPerSecond rampUpPeriodMs Description
1 1 10_000 15_000 2_000 2_000 A single thread and a an amount of max 2000 executions/s over 15 seconds
2 5 10_000 50_000 1_000 2_000 Five threads and max of 1000 executions/s over 50 seconds.
3 10 10_000 150_000 1_000 2_000 Ten threads and max of 1000 executions/s over 150 seconds.
4 15 10_000 300_000 1_000 2_000 Fifteen threads and max of 1000 executions/s over 300 seconds.

Result - Scenario 1 : Binary

Test Case 1

1 Built-in (Java) Converter Function

 Input Expected Output: Threads Execution Time Throughput Min Latency Max Latency Average Latency Invocations Errors
Execution 1 {"data" : "MjA="} digital_input_state = 20 1 15.000 ms 1,192 / s 0.01 ms 0.15 ms 0.02 ms 5,964 0
Execution 2 5 50.000 ms 1,000/ s 0.02 ms 0.69 ms 0.04 ms 40.001 0
Execution 3 10 150.000 ms 1,000/ s 0.01 ms 13,21 ms 0,07 ms 140,000 0
Execution 4 15 300.000 ms 1,000/ s 0.01 ms 6,31 ms 0,06 ms 290,002 0

Test Case 2

Two data points, double Nested Converter Functions (Built-in (Java) + Javascript) Input Expected Output: Threads Execution Time Throughput Min Latency Max Latency Average Latency Invocations Errors
Execution 1 Binary Data with 2 bytes temperature (Byte 1-2), 2 bytes humidity (Byte 3-4) temperature = 20 humidity = 88.19 1 15.000 ms 327/ s 2,45 ms 178,96 ms 3,05 ms 1,638 0
Execution 2 5 50.000 ms 609/ s 3,00 ms 321,31 ms 7,59 ms 24,363 0
Execution 3 10 150.000 ms 604/ s 2,66 ms 387,69 ms 15,01 ms 84,639 0
Execution 4 15 300.000 ms 505/ s 2,42 ms 474,81 ms 26,91 ms 146,481 0

Test Case 3

Double Nested Converter Functions (Built-in (Java) + Javascript) Input Expected Output: Threads Execution Time Throughput Min Latency Max Latency Average Latency Invocations Errors
Execution 1 {"data" : "4f00630063007500700061006e0063007900200002"} sensor_value = 2 1 15.000 ms 512 ms 0,72 / s 159,68 ms 1,46 ms 2,561 0
Execution 2 5 50.000 ms 918 / s 0,69 ms 235,15 ms 1,96 ms 36,735 0
Execution 3 10 150.000 ms 931/ s 0.89 317,7 s 2,26 ms 130,426 0
Execution 4 15 300.000 ms 924 /s 1,15 ms 386,41 ms 3,16 ms 268,218 0

Scenario 2 - Json Data Mapping

Without Converter:

  • Test-Case 1: Simple mapping of temperature value in Json to outdoorTemperature

With JavaScript Converter:

  • Test-Case 2: Mapping and conversion of Json with multiple custom JS functions

With Built-in Converter:

  • Test-Case 3: Mapping and conversion of timestamp in Json into date type sensor value

Each of the tests is executed 4 times with different configuration parameters:

Execution threads warmUpMs durationMs maxExecutionsPerSecond rampUpPeriodMs Description
1 1 10_000 15_000 10_000 2_000 A single thread and a an amount of max 10.000 executions/s over 15 seconds.
2 5 10_000 50_000 10_000 2_000 Five threads and max of 10.000 executions/s over 50 seconds.
3 10 10_000 150_000 10_000 2_000 Ten threads and max of 10.000 executions/s over 150 seconds.
4 20 10_000 300_000 5_000 2_000 Twenty threads and max of 5.000 executions/s over 300 seconds.

Result - Scenario 2:

Test Case 1

No converter functions

 Input Expected Output: Threads Execution Time Throughput Min Latency Max Latency Average Latency Invocations Errors
Execution 1 {"temperature" : 20.3 } outdoorTemperature = 20.3 1 15.000 ms 1,185 / s 0,00 ms 0.13 ms 0.02 ms 5926 0
Execution 2 5 50.000 ms 3,619 / s 0,00 ms 3.84 ms 0.01 ms 144,794 0
Execution 3 10 150.000 ms 4,999/ s 0,00 ms 4,96 ms 0,02 ms 699,937 0
Execution 4 20 300.000 ms 4,999/ s 0,00 ms 9,29 ms 0,01 ms 1,449,991 0

Test Case 2

1 Built-in Converter Function

 Input Expected Output: Threads Execution Time Throughput Min Latency Max Latency Average Latency Invocations Errors
Execution 1 {"time" : 1574736601479} sensor_value = 2019-01-01 15:33:33 +0800 1 15.000 ms 1,196 / s 0,01 ms 0.32 ms 0.05 ms 5,983 0
Execution 2 5 50.000 ms 3,527 / s 0,01 ms 42,92 ms 0.06 ms 141,116 0
Execution 3 10 150.000 ms 4,994 / s 0,01 ms 11,63 ms 0,06 ms 699,284 0
Execution 4 20 300.000 ms 4,996/ s 0,01 ms 21,80 ms 0,06 ms 1,448,878 0

Test Case 3

1 Built-in Converter Function

 Input Expected Output: Threads Execution Time Throughput Min Latency Max Latency Average Latency Invocations Errors
Execution 1 "{\"clickType\" : \"DOUBLE\"}" {"button":{"digital_input_count":2,"digital_input_state":true}} 1 15.000 ms 493 / s 1,32 ms 79,60 ms 1,99 ms 2,469 0
Execution 2 5 50.000 ms 1,290 / s 1,20 ms 253,16 ms 2,99 ms 51,637 0
Execution 3 10 150.000 ms 1,396/ s 1,46 ms 328,80 ms 6,06 ms 195,528 0
Execution 4 20 300.000 ms 1,154/ s 1,58 ms 461,84 ms 15,7 ms 334,853 0