Skip to content

Maestro Data Files

Jump to bottom
Otavio Rodolfo Piske edited this page Jan 17, 2019 · 6 revisions

Data Files

Sender

A sender peer must write the following report files:

  • A file named sender.dat, in the Maestro Binary Rate Format, with the rate recorded during the test.
  • A file named test.properties with the testing properties.
  • A file named system.properties with the system properties.

Receiver

A receiver peer must write the following report files:

  • A file named receiver.dat, in the Maestro Binary Rate Format, with the rate recorded during the test.
  • A file named test.properties with the testing properties.
  • A file named system.properties with the system properties.
  • A file named receiverd-latency.hdr with the latency distribution recorded during the test.

Inspector and Agent

Other types of peers may provide files in any format suitable for the operation they perform. Ideally, they should provide conversion tools to make them accessible via the reports server.

File Formats

Maestro Binary Rate Format

The Maestro Binary Rate Format is used to store the rates recorded during the test. The format consists of the following fields: header and records.

Header Records

Header

The header contains metadata related to the file format and contents. It consists of the following fields: format name, format version, maestro version and role.

Format Name Format Version Maestro Version Role
  • Format Name: a string with the text "maestro", used to identify the file format.
  • Format Version: an integer containing the format version, used to identify the version of the data file. The current version is 1.
  • Maestro Version: an integer containing the Maestro version used to create the file. The number should be the version string stripped of non-numeric chars. For example, 1.5.0-SNAPSHOT becomes 150, 1.5.1 becomes 151, and so on.
  • Role: the numeric peer role code as an integer as documented in the Maestro Orchestration Protocol.

Records

The records section of the file consists of one or more records. A single record consists of the following fields: metadata, count and timestamp.

Metadata Count Timestamp
  • Metadata: record metadata. Currently reserved for the future. It should be 0 by default for the current version. It it an integer.
  • Count: a long integer containing the number of messages exchanged within 1 second.
  • Timestamp: a long integer containing the time of collection, with microseconds precision.

Test Properties File

Contain the test properties as setup by a client.

Sample content for the test.properties:

#maestro-plotter
#Tue Jan 15 12:31:52 UTC 2019
fcl=0
parallelCount=1
apiName=JMS
duration=1000000
limitDestinations=1
durationType=count
rate=0
variableSize=false
apiVersion=1.1
brokerUri=tcp\://sut\:61616/test.performance.queue?protocol\=OPENWIRE&durable\=true&limitDestinations\=1
messageSize=100
protocol=OPENWIRE

System Properties File

Contain the system properties of the peer running the test. Sample:

#maestro-plotter
#Tue Jan 15 12:31:52 UTC 2019
workerJvmVersion=25.191-b12
workerOperatingSystemName=Linux
workerOperatingSystemCpuCount=1
workerOperatingSystemVersion=3.10.0-957.el7.x86_64
workerOperatingSystemMemory=4026138624
workerJvmName=OpenJDK 64-Bit Server VM
workerOperatingSystemArch=amd64
workerJvmJavaHome=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre
workerJvmJavaVersion=1.8.0_191

HDR Latency File

One of the files generated by the receiver, contains the latency information. That file is named receiverd-latency.hdr and it is used to store the receiver latency data.

The latency data in stored in the compressed HDR Histogram log format. Because this is managed by a third party library, it is recommended to check the format details that are available here, and here and here.

A compliant file is provided below as a sample:

#[mpt]
#[Histogram log format version 1.3]
#[StartTime: 1547555512.260 (seconds since epoch), Tue Jan 15 12:31:52 UTC 2019]
"StartTimestamp","Interval_Length","Interval_Max","Interval_Compressed_Histogram"
1547555512.260,1799.954,27.853,HISTFAAACw142j1YPY9lRxHtqVu3br++/Xp63rx5M7uz62G9GLNarA1WaLUCtDIIWciykIUIkOWAyBEhPwAJAuINCAgIQCIgQCT8A4sfgAQRITkJOeecvsOO/d67/VFdH+dUVd9Hv3x7mdLVX9P4N23fZ/z4x9s/pDf/GQNfLvbW7Mq+Z3t/aec24/e53dkzs8kufLbFF7uwhGFbbWcf2IRBjGDlhL+Z6y4xMWNs8dXtmlM+2YJNGp/wI/HTz31yrl8cH8bF+N4bF85DllnY/5fv9MvsxZj1Pb53Pm0izw2aacXsfF5jN45yLEimub1PksTHhN8rdJ9hh1SnIA7O3GBaqQOg0RTcY0MPWDx+8wlir/Vzm7JhiV3qVD+H8Mk8bUugyxNupd0QzDlaJLMo695CKky9YCJOu+SA5jQptWbTkntfXnqS6y7pPvyjK7BOn3KJ3K/HW0iC6CT9FJnHGIFz49Jk2p0kUoW9nLWF7hohn6UAll9u84z0Tqrd6mRGnTHe63NmXC+wKBEgcPdAQoJZ1/gmztah5rqF0uwpnnjubhy2k9BVEX1hQ1fE63zYssKdz6D0Cu3g0jtF8XoI5MmPYRaXzn6FSQzdQZkfSKFbHE793zMMPrFXkEy1+T+xxP0rnPEUvxO0pbtgtn/LdhohIz62B/bGPrGdBGPLUxz0eiBtsQ/JD6dKV5CdsP4lPhF1HPkFdOPhoAj+7jDyMXT4FCvorceS/T5W//oM1rzE0STiJ/Yjt9dYfWG/OoMIGCIXAOqY/wITP8XGX5xBn99RmwUiPrKfQZEZUr6jqD6x357Zz7H8uf3tDEe9tv+afWa/OcPAh3i+wko41H5v9ucVs3/f2b/DvpxNGPiL2x/P5PR/nomPF9jwhmEkrL4vp/2Johjvz2HHBQYfKGI/wSZ68xZZBQtm6QOa2A8VM+SRHXMMcbKzH0PiN6EJTQV7wKIPiAX6ayWoEs0bcgmUB/Cl24ApUJCCiUUx2IkFxO459roLwiLEAmn+EeQq3TB0q8CRJPV6cC0g1lIe1MYpQXwgW0FjZa5ZnMO6O45fwwfcPmWXkAQ7r3n+ikBsh07hEnal52tmBhF9lhIvoOZ72LX64hutdjiAKDnXQSAFELUIca9sIO0CYhYBYbBD8OX5Kxdc0+ZdXDldJ9oyNSwbPZkiXTljhvX+BDulGDcjV5FGa0BHf59rVl9H8hn0EqnFNX8lDZjfYhoeZkSUvRfG1DU3bcloRx+u3K30azHjL43C8UC8TT7yLMfOlbHI35munHwAZlOGARojSnOx85FBiUlX1sRf3uHA81FPOAvzPTMeGCRgUBcwcr15zpEWt5wrA4k4lTzVNgrZD7v3SnfTSPCaHfluol1aTnbfukAKJwJA0DBUD11gVXobyu9LinsE06aaBgZHuaLMyYP+gmu0gvmycLPTB0H7yyS9LGfYwP/4j8JSZdzq4vLR0CRRIma2OnFfojLdljKDT+/yQcUvN9aGnIoFxGBjyfhsFV+Rm5UQH6hYwIJgVIYpOwIIA+3gow47gTfqWaZvgtXWt+A4z/JRARe5aq9qCwNS/obRLNU3LKkZaIvGobIoTyTa51CAWkAif0CHnAeeWPBCDh3BoooHVvocJjdGlIqdC9VNo8AWF6WZVfLE0SwExSZHYAnY4UWwiwE+xQRqJd+imGUa/UkPHKS/A4MWJaxAsKDjo/Z7VZzu1ad78rAOAirapIAZ+A+JSW43wCCrueBmrhzHJBPxRlzogGDMY6DKxylcI89EKsqngE5YkMcyNBcTYqCeZ8dxhAYeYSHUQPCNKFxExDmwLDUPnkSf2iB7lmqwVHgAOLG7UCKlDUoIA3mkiYjR6ABUst9ztdw950mWQTxXIlpwBgNDkGMFPZHlWDuEOJIH2/pEX+FEABU/IDW3hnanJPoVjxD2yE5WctZA4nfpwfNFuig8+tANphXPeES4ew0aElGpL8XnDjLVIzYFho6KAlyE2Q73hqKHT8xXPgXsAnduvm4NTu7grjcgGr+dyCgRmfyocH5tqYOYuZF9R2Y9mFXE7SiiUT4ICK0SZ9kbJNR6Uz0OHtAIIT3BgQW+a5CKGAS3lwAzsKJiQ8Lp1Xt7VPPxACObIA/MF5pjtM9byuUAtngGECYc2aE4dE49Pwxam3sw8tieYSKIxXCB4V1hZVoppeaWGXH4TkmYIaNb9FXgG3EcrlQ0J84wilh5HKkmURIMwsLAt9Mg7KggZbWGfJBRsluB4/OJ5wgrUeh3rWtIHRVR9VbyDUzBWCFa6qECaVAuE22kgXcm0negHTNcrYwiuUd3UO/SB42JSUISWTumkpR9aM5oIkiYTC0LsW+deufhRDCvlKY80An0INw6tCK2rODkQJbL5BmxArRkcrGXE9CH3Aw3Jx/pnkEafspMv2QL2U9KRYhZrUENBsHo4pLocSfKaBRXVnKQPEqMKWkl4uOAYDCsIeoEe9QTVk0MF1Az0d9YAN0hqpkyOOFLr5aeS50NnsPZMDF6ru1RR2BuKpiDFDS9S1LB65Xof8hDgwe2m1JLOXE4xSHnfARNANhoGTKP0cg/+PFw47Ue/N0OJFf6IVsLpiWs7DkdH8KbTH/tBmDpNxDgwHb0djwCBQGcQNnjO3Yib8uhH6BAboVq0NeIeofvYSqlUj6jD090+nUCy+ijBhpgsx2PJ+w4dlhaC0oTQwZQEpKlA3Zgc6lEfVNFbEV0ZVJk6CLX3ou4gZJVgHs4C7jEwNG7PJ8A0a8iP9nBjZmGxxSuF+WBP+IyEQYhw5CT8IzigBAw2ox6VgKtpAKrRkUaoDfBBiX8UBk6JPqdaCrYzsKRJ9QeKAYSYVslXCbyMITzPJP4/Af3ElFQlh0IktVhMZZVlPM8qWCKjc4WOqmazpW3Qh3bk8ykZUr9JhapKxi13SjZlE1hxTzaAfADNoF4CY2kz1v1RF7Up289BTUy9WGzcoirqG3vH2a+2uCuwzS2WB7FPHRnzSyhXd3WaMDYvURZRpFkHmrzqFkLa4Y6KN4O1rrGuKGjD6Lkk3pIlmcIZS/Eyo0x1JSkdMdeLxgx38el1uWDaQf+OjsLNgsIVVYfH+PNhm+H0LCp0VsTCIcBoC+P9s5GSvO2Y/G+keuQUSGsKE4z0/QiIap5TN5qkuhzH9BKEL1jFi2jeqvh71Tf9eZjzrE0FiR0RTBhztPha0oto/M66b1J/fZ32dqplXYWV76GoHXsM0PwSExsNlplIYyVdK4nwsKXXNTsZBxB9ZieqWoZN5AkugPvVkZzsymg5gzeAcKzqgVdBhpzDqxlYxQkUmdA4GBCI9TFMkcuoCCB9DAOwsUJiTuNiobG5mERSoHyYhoGOyqzE+zrrBgs3DCspZuyAQbmgA6n0S8jRSbkC9Zhgn/yplaKPlDjybKmQgnoVTYFLCLWwek0jKTNpNsRKpPeBPUR/G/G5ofURGO30N1uijXrrVIZ5vNUWctG15XHlWImN1nF0cjASo0wGsQsHZNH5+tXWT2dulB5H5cY963nw/7Cpq0QvjOPCt0H5ptRIXkf8TKTGHA/MwjqJ5GkhDnxcgJ8ZV3w8sAiclFe+3gNNshr43VYqhvHdZtBZWSx1m0TzlyCVzvcf11XLlzbMi+GCJWPl4E8hF14LqjfX0EskugbJPtyfxntx1QGQ5WJmDYYPjiKd5R9W9Q036rlXrOuVXxBgllMLPEgcGVd/UrvTXU/veg8fUflyqoENK65s50DhHEpOqIrrzye75Fe+qJ3rrM97s/qc39s/zpzex7/Azsvk24=

Inspector Files

The broker inspector peers should generate a compressed CSV file (they should be named brokerd-rate.csv.gz). This file should store SUT specific telemetry data.

The broker inspector file is composed of the following columns:

  • timestamp: the date and time for the data sample in the format %Y-%m-%d %H:%M:%S (according to Glibc's strftime specification) or YYYY-MM DD hh:mm:ss using the W3C defined standard for datetime.
  • load: system load
  • open fds: number of opened filed descriptors.
  • free fds: number of free file descriptors.
  • free mem: free physical memory.
  • swap free: swap free memory.
  • swap committed: swap committed memory.
  • eden inital: eden initial memory.
  • eden committed: eden committed memory.
  • eden max: eden maximum (limit) memory.
  • eden used: eden used memory.
  • survivor inital: survivor initial memory.
  • survivor committed: survivor committed memory
  • survivor max: survivor maximum (limit) memory.
  • survivor used: survivor used memory.
  • tenured inital: tenured inital memory.
  • tenured committed: tenured committed memory.
  • tenured max: tenured max memory.
  • tenured used: tenured used memory.
  • pm inital: permgerm or metaspace initial memory (either permgen or metaspace depending the JVM version).
  • pm committed: permgerm or metaspace committed memory (either permgen or metaspace depending the JVM version).
  • pm max: permgerm or metaspace maximum memory (either permgen or metaspace depending the JVM version).
  • pm used: permgerm or metaspace used memory (either permgen or metaspace depending the JVM version).
  • queue size: number of messages waiting for processing on the queue.
  • consumers: number of consumers connected to the queue.
  • ack: number of acknowledged messages on the queue.
  • exp: number of expired messages on the queue.

Sample file:

"timestamp","load","open fds","free fds","free mem","swap free","swap committed","eden initial","eden committed","eden max","eden used","survivor initial","survivor committed","survivor max","survivor used","tenured initial","tenured committed","tenured max","tenured used","pm initial","pm committed","pm max","pm used","queue size","consumers","ack","exp"
"2017-10-05 15:06:31",0.0,634,3462,11828,11828,14427,256,340,1364,13,42,0,0,0,683,1162,2731,927,0,43,17592186044415,42,44,4,250,0
"2017-10-05 15:06:41",0.0,634,3462,11829,11829,14427,256,340,1364,167,42,0,0,0,683,1162,2731,927,0,43,17592186044415,42,51,4,32747,0
"2017-10-05 15:06:51",0.0,634,3462,11828,11828,14427,256,340,1364,319,42,0,0,0,683,1162,2731,927,0,43,17592186044415,42,22,4,65158,0
Clone this wiki locally