@@ -315,8 +332,8 @@
Bug Fixes
-- Fixed bug with **Logstash** in
experimental mode that didn't process **WebRTC traces** correctly.
-- Fixed bug by running **`several TJobs with browsers`** that made them fail.
+- Fixed bug with **Logstash** in
experimental mode that didn't process **WebRTC traces** correctly.
+- Fixed bug by running **`several TJobs with browsers`** that made them fail.
@@ -327,15 +344,15 @@
Features & Improvements
-- Now you can configure a Sut with parameters in
ElasTest Jenkins Plugin.
-- Now when you specify the use of a
Sut with external Elasticsearch, you can also set the _`path`_ in which the service is located, if necessary.
-- A new
environment variable has been added to the TJob execution containers: **`ET_NETWORK`**.
-- **Containers** now contain **`labels`** with information to be used by ElasTest.
+- Now you can configure a Sut with parameters in
ElasTest Jenkins Plugin.
+- Now when you specify the use of a
Sut with external Elasticsearch, you can also set the _`path`_ in which the service is located, if necessary.
+- A new
environment variable has been added to the TJob execution containers: **`ET_NETWORK`**.
+- **Containers** now contain **`labels`** with information to be used by ElasTest.
Bug Fixes
-- Fixed the bug that appeared randomly when running a TJob with EUS
-- **ET_SUT_CONTAINER_NAME**
environment variable was not being sent to
Sut in new container.
+- Fixed the bug that appeared randomly when running a TJob with EUS
+- **ET_SUT_CONTAINER_NAME**
environment variable was not being sent to
Sut in new container.
@@ -346,16 +363,16 @@
Features & Improvements
-- We've made improvements to the
Documentation.
-- We have improved
example projects and added new ones. The technologies used are
Java JUnit5,
Java JUnit4,
Python,
JS Jasmine,
JS Protractor,
Java Karate,
Java Cucumber and
Java Gauge.
-- It is no longer necessary to specify the **`--server-address=localhost`** when running on
MAC or
Windows (In docker for Windows, in Docker toolbox **`--server-address=$(docker-machine ip)`** is still necessary).
-- All active live sessions are now shown in
Web Browsers section.
-- A
button to restore the demo projects has been added on the project card.
--
ElasTest Jenkins Plugin has been updated to
Java 8.
-- A system of **retries** has been added to send the traces to ElasTest in the
ElasTest Jenkins Plugin.
-- The use of **Suts managed by ElasTest** is allowed now in
ElasTest Jenkins Plugin.
-- **Logstash** has been updated to 6.5.0 version (Logstash is only available in Elastest
experimental mode)
-- Now the tests can use a
single browser for all the tests but the videos are associated with each test.
+- We've made improvements to the
Documentation.
+- We have improved
example projects and added new ones. The technologies used are
Java JUnit5,
Java JUnit4,
Python,
JS Jasmine,
JS Protractor,
Java Karate,
Java Cucumber and
Java Gauge.
+- It is no longer necessary to specify the **`--server-address=localhost`** when running on
MAC or
Windows (In docker for Windows, in Docker toolbox **`--server-address=$(docker-machine ip)`** is still necessary).
+- All active live sessions are now shown in
Web Browsers section.
+- A
button to restore the demo projects has been added on the project card.
+-
ElasTest Jenkins Plugin has been updated to
Java 8.
+- A system of **retries** has been added to send the traces to ElasTest in the
ElasTest Jenkins Plugin.
+- The use of **Suts managed by ElasTest** is allowed now in
ElasTest Jenkins Plugin.
+- **Logstash** has been updated to 6.5.0 version (Logstash is only available in Elastest
experimental mode)
+- Now the tests can use a
single browser for all the tests but the videos are associated with each test.
Bug Fixes
- Now the
environment variables ET_SUT_PORT and ET_SUT_PROTOCOL are passed to the test container.
@@ -416,19 +433,19 @@
1.0.0-beta1
12/09/2018
--
Improvement in requirements. We've been working on the high memory consumption requirements of previous versions, and this new version can be deployed on machines with 8Gb of RAM starting ElasTest in default mode
+-
Improvement in requirements. We've been working on the high memory consumption requirements of previous versions, and this new version can be deployed on machines with 8Gb of RAM starting ElasTest in default mode
--
GUI improvements. Among other small improvements, it should be noted that more information is now shown to the user, such as the percentage of downloading a docker image, either from a tjob, a sut or a browser.
+-
GUI improvements. Among other small improvements, it should be noted that more information is now shown to the user, such as the percentage of downloading a docker image, either from a tjob, a sut or a browser.
--
Improvements in Jenkins plugin. It is no longer necessary for the user to start dockbeat and filebeat to send logs and metrics to ElasTest, the plugin is responsible for it now.
+-
Improvements in Jenkins plugin. It is no longer necessary for the user to start dockbeat and filebeat to send logs and metrics to ElasTest, the plugin is responsible for it now.
--
Jenkins instance provided by ElasTest. You can use now a Jenkins instance provided by ElasTest, can be started on load ElasTest or later manually from the ElasTest GUI.
+-
Jenkins instance provided by ElasTest. You can use now a Jenkins instance provided by ElasTest, can be started on load ElasTest or later manually from the ElasTest GUI.
--
Improvements in TestLink integration. It is possible to configure a Sut so that ElasTest starts it every time a Test Plan is executed
+-
Improvements in TestLink integration. It is possible to configure a Sut so that ElasTest starts it every time a Test Plan is executed
--
Improvements in log analysis.
+-
Improvements in log analysis.
--
Bug fixes.
+-
Bug fixes.
@@ -437,22 +454,22 @@
0.9.1
09/05/2018
--
Jenkins integration. You don’t need to abandon your CI tools. Use ElasTest straight from your Jenkins jobs (Freestyle or Pipeline). For instance, if you’re using Pipeline jobs, just wrap up your Pipeline in an elastest step, and you’ll be ready to use all the fancy features we bring to testers and developers, like managed browsers! See our advanced example for a more complex scenario.
+-
Jenkins integration. You don’t need to abandon your CI tools. Use ElasTest straight from your Jenkins jobs (Freestyle or Pipeline). For instance, if you’re using Pipeline jobs, just wrap up your Pipeline in an elastest step, and you’ll be ready to use all the fancy features we bring to testers and developers, like managed browsers! See our advanced example for a more complex scenario.
--
TestLink integration. TestLink is the most widely used web-based test management tool and ElasTest provides integration with TestLink. With this integration now you can easily record videos and gather logs and metrics when manually running your tests.
+-
TestLink integration. TestLink is the most widely used web-based test management tool and ElasTest provides integration with TestLink. With this integration now you can easily record videos and gather logs and metrics when manually running your tests.
--
New Docker images for browsers. We have been developing a new set of browser Docker images that brings the benefits of both Selenoid and Selenium docker images. These images are ready for some awesome features we will be developing in the following months, like changing browser resolution to adapt better to the desktop of our users.
+-
New Docker images for browsers. We have been developing a new set of browser Docker images that brings the benefits of both Selenoid and Selenium docker images. These images are ready for some awesome features we will be developing in the following months, like changing browser resolution to adapt better to the desktop of our users.
--
Browser recordings can now be opened as a new tab or on a floating dialog.
+-
Browser recordings can now be opened as a new tab or on a floating dialog.
--
Our powerful Log Analyzer now remembers previous configurations.
+-
Our powerful Log Analyzer now remembers previous configurations.
--
Generate your SuT on the fly with Commands Containers: a new Docker image can be generated on the fly, and this image will be run as the SuT.
+-
Generate your SuT on the fly with Commands Containers: a new Docker image can be generated on the fly, and this image will be run as the SuT.
--
ElasTest Command line options documented in its own subsection within the installation page.
+-
ElasTest Command line options documented in its own subsection within the installation page.
--
Changes in environment variables: the ElasTest endpoint for sending metrics from an external SuT has changed, and the ET_MON_LSHTTP_API environment variable is now available also for Test Support Services.
+-
Changes in environment variables: the ElasTest endpoint for sending metrics from an external SuT has changed, and the ET_MON_LSHTTP_API environment variable is now available also for Test Support Services.
--
There are new environment variables available: ET_SUT_CONTAINER_NAME, ET_MON_INTERNAL_LSBEATS_PORT and ET_SUT_LOG_TAG. Those allows ElasTest to do a better job when identifying what’s coming from which SuT.
+-
There are new environment variables available: ET_SUT_CONTAINER_NAME, ET_MON_INTERNAL_LSBEATS_PORT and ET_SUT_LOG_TAG. Those allows ElasTest to do a better job when identifying what’s coming from which SuT.
-- We have changed the name of the status for a TJob that has been stopped manually to STOPPED.
+- We have changed the name of the status for a TJob that has been stopped manually to STOPPED.
diff --git a/docs/tutorials/using-browsers-in-elastest.md b/docs/tutorials/using-browsers-in-elastest.md
index d96d263..31f5114 100644
--- a/docs/tutorials/using-browsers-in-elastest.md
+++ b/docs/tutorials/using-browsers-in-elastest.md
@@ -139,11 +139,11 @@ public class ElastestBaseTest {
}
```
-> - **`ET_SUT_HOST`**, **`ET_SUT_PORT`** and **`ET_SUT_PROTOCOL`** variables will be the IP, port and protocol of our SuT respectively. ElasTest will automatically inject the right value (Know more about
Environment Variables)
+> - **`ET_SUT_HOST`**, **`ET_SUT_PORT`** and **`ET_SUT_PROTOCOL`** variables will be the IP, port and protocol of our SuT respectively. ElasTest will automatically inject the right value (Know more about
Environment Variables)
-> - **`ET_EUS_API`** variable tells us where to connect to use Elastest browsers (standard Selenium Hub). If the variable has no value, we can consider that this service is no available and then local browsers have to be used (here we are using
WebDriver Manager Java library. This library is responsible to download and configure any additional software needed to use installed browsers from tests)
+> - **`ET_EUS_API`** variable tells us where to connect to use Elastest browsers (standard Selenium Hub). If the variable has no value, we can consider that this service is no available and then local browsers have to be used (here we are using
WebDriver Manager Java library. This library is responsible to download and configure any additional software needed to use installed browsers from tests)
-> - The values of the variables **browserType** and **browserVersion** are taken from the **properties** browser and browserVersion respectively, which you can pass in the test run command with **`-Dbrowser=chrome`**.
+> - The values of the variables **browserType** and **browserVersion** are taken from the **properties** browser and browserVersion respectively, which you can pass in the test run command with **`-Dbrowser=chrome`**.
#### **Jenkins Pipeline**
@@ -166,7 +166,7 @@ The example above can be split into the following sections:
-- **ElasTest plugin block with configuration** : this block will contain all the steps that the test must follow, as well as the necessary [configuration options](#options). In this case is very important the part **`tss: ['EUS']`** because allow us use the browsers.
+- **ElasTest plugin block with configuration** : this block will contain all the steps that the test must follow, as well as the necessary [configuration options](#options). In this case is very important the part **`tss: ['EUS']`** because allow us use the browsers.
@@ -180,7 +180,7 @@ node{
-- **Test Execution** : Dowload the git project and execution test, we change the type of browser with the variable **`-Dbrowser=`**, and define the version with the variable **`-DbrowserVersion=`**
+- **Test Execution** : Dowload the git project and execution test, we change the type of browser with the variable **`-Dbrowser=`**, and define the version with the variable **`-DbrowserVersion=`**
@@ -339,7 +339,7 @@ public class FullTeachingTestE2EREST extends FullTeachingTestE2E {
We can observe the all code in this [link](https://github.com/elastest/full-teaching-experiment/blob/master/src/test/java/com/fullteaching/backend/e2e/FullTeachingTestE2EREST.java).
-> - The test mark the init of the test case in the **`setup`** method and the finish of the test case in the **`dispose`** method. With the code `log.info("##### Start test: " + info.getTestMethod().get().getName());`and `log.info("##### Finish test: " + info.getTestMethod().get().getName());` respectively.
+> - The test mark the init of the test case in the **`setup`** method and the finish of the test case in the **`dispose`** method. With the code `log.info("##### Start test: " + info.getTestMethod().get().getName());`and `log.info("##### Finish test: " + info.getTestMethod().get().getName());` respectively.
```java
@ExtendWith(SeleniumExtension.class)
@@ -372,7 +372,7 @@ public class FullTeachingTestE2E {
We can observe the all code in this [link](https://github.com/elastest/full-teaching-experiment/blob/master/src/test/java/com/fullteaching/backend/e2e/FullTeachingTestE2E.java).
-> - **`ET_SUT_HOST`**, **`ET_SUT_PORT`** variables will be the IP of our SuT respectively. ElasTest will automatically inject the right value (Know more about
Environment Variables)
+> - **`ET_SUT_HOST`**, **`ET_SUT_PORT`** variables will be the IP of our SuT respectively. ElasTest will automatically inject the right value (Know more about
Environment Variables)
#### **Jenkins**
@@ -434,7 +434,7 @@ def containerIp(containerName, network) {
The example above can be split into the following sections:
-- **ElasTest plugin block with configuration** : this block will contain all the steps that the test must follow, as well as the necessary [configuration options](#options). In this case is very important the part **`tss: ['EUS']`** because allow us use the browsers.
+- **ElasTest plugin block with configuration** : this block will contain all the steps that the test must follow, as well as the necessary [configuration options](#options). In this case is very important the part **`tss: ['EUS']`** because allow us use the browsers.
@@ -448,7 +448,7 @@ node{
-- **Sut configuration** : The SUT must be started, passing the **`${env.ET_SUT_CONTAINER_NAME}`** env variable (provided by ElasTest) as name of the container. This will allow ElasTest to receive logs and metrics from the Sut.
+- **Sut configuration** : The SUT must be started, passing the **`${env.ET_SUT_CONTAINER_NAME}`** env variable (provided by ElasTest) as name of the container. This will allow ElasTest to receive logs and metrics from the Sut.
@@ -461,8 +461,8 @@ sutIp = containerIp(sutContainerName,network)
-- **Wait for Sut** : You have to obtain the Sut network and ip and run check image (elastest/etm-check-service-up) provided by ElasTest to wait for the Sut to be ready to be used. This step is not required, you can wait in other ways or not do it, but for this example we do it.
-
+- **Wait for Sut** : You have to obtain the Sut network and ip and run check image (elastest/etm-check-service-up) provided by ElasTest to wait for the Sut to be ready to be used. This step is not required, you can wait in other ways or not do it, but for this example we do it.
+
```groovy
echo 'Sut ip: '+ sutIp
@@ -497,7 +497,7 @@ def containerIp(containerName, network) {
-- **Test Execution** : Finally, the tests should be executed to verify that the SUT is working correctly. Remember that you have to configure the Sut ip as an environment variable or pass it as a maven property so that the test knows where the SUT is.
+- **Test Execution** : Finally, the tests should be executed to verify that the SUT is working correctly. Remember that you have to configure the Sut ip as an environment variable or pass it as a maven property so that the test knows where the SUT is.
@@ -512,7 +512,7 @@ stage("Run Tests") {
-- **Stop Sut** : The SUT will stop when the test is finished, passing the **`${env.ET_SUT_CONTAINER_NAME}`** env variable (provided by ElasTest) as name of the container.
+- **Stop Sut** : The SUT will stop when the test is finished, passing the **`${env.ET_SUT_CONTAINER_NAME}`** env variable (provided by ElasTest) as name of the container.
@@ -594,7 +594,7 @@ Elastest can also start browsers on an **AWS** instance. For this, it is necessa
**`amiId`** and **`region`** values **must** be as above.
-We could **adapt** the code of our **`ElastestBaseTest.class from the first example`** to use browsers in **AWS**. For this we have added the **`initAwsConfig`** method that is called in the *@BeforeAll* and initializes the awsConfig object from values that can be sent in environment variables. Afterwards, you only need to add the awsConfig object to a **capability**.
+We could **adapt** the code of our **`ElastestBaseTest.class from the first example`** to use browsers in **AWS**. For this we have added the **`initAwsConfig`** method that is called in the _@BeforeAll_ and initializes the awsConfig object from values that can be sent in environment variables. Afterwards, you only need to add the awsConfig object to a **capability**.
##### **ElastestBaseTest class**
@@ -747,3 +747,49 @@ public class ElastestBaseTest {
###### **Demo**
You can see a demo
here.
+
+
QoE testing with EUS
+
+Quality of Experience
(QoE) is a measure of the delight or annoyance of a customer's experiences with a service. EUS offers an api for calculating QoE metrics for video, which is very useful for testing quality in videoconferencing applications such as
OpenVidu.
+
+The QoE test offered by EUS consists of the comparison of two videos: the first video would be the original video published by an endpoint and the second the video received through that endpoint. This comparison generates several csv with different quality metrics. You can take a look at these
example results.
+
+This functionality is available since version 2.1.0 of ElasTest, and a
sample demo project is included. Is only available in elastest mini and siglenode modes, for Docker and AWS. For Kubernetes (EK and HEK) is not available yet
+
+In order to calculate qoe metrics, it is **necessary** that the videos to be processed (original video + received video) have a **padding at the beginning and at the end**. For this you can use
this script. It is recommended to set a padding of at least 12 seconds, replacing the default value of the variable PADDING_DURATION_SEC (5).
+
+Once you have obtained the original video (.y4m) and audio (.wav) with padding (two files are generated automatically), you must **include it as a fake resource capability** in your test, so that instead of showing the video from the camera this video is used.
+
+```
+ ChromeOptions options = new ChromeOptions();
+ // This flag avoids to grant the user media
+ options.addArguments("--use-fake-ui-for-media-stream");
+ // This flag fakes user media with synthetic video
+ options.addArguments("--use-fake-device-for-media-stream");
+ // This flag allows to load fake media files from host
+ options.addArguments("--allow-file-access-from-files");
+ // This flag sets the video input (with padding)
+ options.addArguments("--use-file-for-fake-video-capture=/path/to/video_with_padding.y4m");
+ // This flag sets the audio input
+ options.addArguments("--use-file-for-fake-audio-capture=/path/to/audio_with_padding.wav");
+
+```
+
+The browser must be able to access these files. To do this EUS has an **api** to be able to **upload these files** during the execution to the browser container. If the files are accessible from the test you can simply attach them to the call, but you can also have them in an online repository and send the url to the request and the EUS itself download it into the browser container, which is exactly what is done in the above mentioned
sample demo project.
+
+To do this, call to the EUS api with path **`/browserfile/session/{sessionId}/`** and the following parameters:
+
+- **fileUrl**: the remote and accesible url where the video is stored.
+- **fileName**: the name under which the video will be saved inside the browser container. Should be the same as that defined in **`--use-file-for-fake-video-capture`**
+- **filePath**: the path under which the video will be saved inside the browser container. Should be the same as that defined in **`--use-file-for-fake-video-capture`**
+
+Then do the same for the audio file.
+
+To start the test call the EUS api (/session/{sessionId}/webrtc/qoe/meter/start) with the following parameters:
+
+- **presenterPath**: the path of the original video in the container of the browser that emits it.
+- **presenterSessionId**: the session id of the browser that broadcasts the original video.
+- **viewerPath**: the path of the received video in the container of the browser that receives it.
+- **viewerSessionId**: the session id of the browser receiving the broadcast video.
+
+This request is in charge of **initiating the service** that is in charge of calculating the QoE metric, obtaining and passing to this service the videos of the browser containers. Then it executes in the **background the calculation** of the metrics and it will **return the Id of the service** as an answer, so that from the test the EUS can be polled to know the state of the calculation. Once completed, the EUS itself will obtain the **csv generated and attach them** to the execution. It will also calculate the **average of each metric and attach** it.
diff --git a/mkdocs-web.yml b/mkdocs-web.yml
index ae99637..82fcbbe 100644
--- a/mkdocs-web.yml
+++ b/mkdocs-web.yml
@@ -31,6 +31,7 @@ pages:
- 'Cross browser testing': 'demos/cross-browser-testing.md'
- 'Security testing': 'demos/security-testing.md'
- 'Chaos testing': 'demos/chaos-testing.md'
+ - 'QoE testing': 'demos/qoe.md'
- 'Test recommendation': 'demos/test-recommendation.md'
- 'Test cost estimation': 'demos/test-cost-estimation.md'
diff --git a/mkdocs.yml b/mkdocs.yml
index 7e63ebc..6adcd33 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -31,6 +31,7 @@ pages:
- 'Cross browser testing': 'demos/cross-browser-testing.md'
- 'Security testing': 'demos/security-testing.md'
- 'Chaos testing': 'demos/chaos-testing.md'
+ - 'QoE testing': 'demos/qoe.md'
- 'Test recommendation': 'demos/test-recommendation.md'
- 'Test cost estimation': 'demos/test-cost-estimation.md'
+ 
+ 
+ 
+
+ 
