Current coverage value always lower than baseline coverage value [was: Different baseline coverage data]

[fsdschmidt]

We have some trouble with coverage comparison. For example our current build successfully detected a baseline build:

Found baseline build with ID 97635.
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 75747
Covered lines: 37405
Code Coverage (%): 49.3815

Unfortunately the detected baseline code coverage % ist wrong, to high and above our variance. Here is the quality check of baseline build with ID 97635:

Validating code coverage policy...
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 75641
Covered lines: 36932
Code Coverage (%): 48.8254
Found baseline build with ID 97599.
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 75747
Covered lines: 37408
Code Coverage (%): 49.3855

For any reason the detected coverage in a later comparison build is higher than the original one. And also total lines differ. TFS reports a coverage of 49.3 % (summary page), may be 48.8 % of the current build is wrong? We use line coverage.

[ReneSchumacher]

Hi @fsdschmidt,

could you post the configuration of the Build Quality Checks task here or send it to PSGerExtSupport@microsoft.com so we know exactly how coverage evaluation is configured? And do you have multiple test runs in your pipeline? If you do, Azure DevOps aggregates coverage values (at least from the VSTest task) so the result might depend on the time you read the value from the service.

Coverage values shown in the Code Coverage tab of the build summary can differ from the values our policy shows. It should only happen for non-VSTest test runs though. I know that there is a bug in the Cobertura report generation of IstanbulJS. Thus, if you're running tests for JavaScript/TypeScript, you will most likely see different coverage values in the Code Coverage tab.

If you like you can also send us the logs from the baseline build and the current build so we can dig deeper.

René

[fsdschmidt]

Hi René,

thanks for your fast reply. I will offer some details of our build configuration (TFS without yaml). We have multiple test tasks in two parallel agents:

Agent 1:
VSTest Server
.NET Core Server Unit
.NET Core Server Integration

Agent 2:
VSTest Client
.NET Core Client

After completion of both build agents we have one final agent which performs the test quality check for whole build:

Which logs would be interesting for an deeper analysis?

[ReneSchumacher]

Hi again,

sorry for the delay. Just to confirm my understanding: you have one build pipeline with three agent jobs:

1. job builds and tests the server code
2. job builds and tests the client code
3. job is dependent on the server and client jobs and runs the Build Quality Checks task

Is that correct?

While this setup should work, we've never explicitly tested it. Is this an intermittent issue or does it consistently occur? To fully understand what's going on I would need the logs of all three builds (the "current" build from your first post, the baseline of the current build with ID 97635, and the baseline of the baseline build with ID 97599). It would also help if you ran your builds with the variables System.Debug and BQC.LogRawData set to true. With those variables set, the JSON data from the different API calls is written to the build log so we can see the exact data our task is relying on.

Since those logs may contain sensitive data (e.g., URLs of your TFS instance, project names, etc.) I'd recommend sending them via email to PSGerExtSupport@microsoft.com and/or removing sensitive information from the logs.

[fsdschmidt]

Yes this is correct. For faster builds we run server and client build in parallel. But afterwards we want to have only one coverage info, so that's job of agent 3.

The issue occurs consistently on our master branch and the related build. It works on further branches and builds, but at this point I don't understand why.

For further analyses I´ve sent some logs via email.

[ReneSchumacher]

This issue was caused by the asynchronous processing of coverage data in Azure DevOps paired with an issue in the processing jobs causing them to report the wrong status for the backend process. Due to this wrong status, the Build Quality Checks task reads premature coverage data that hasn't been fully aggregated, causing it to use only part of the coverage data for policy evaluation.

If BQC is always reporting lower coverage than there actually is, you can add a task that runs before BQC to delay policy evaluation. The simplest workaround is adding a PowerShell task with the following inline script: Start-Sleep -Seconds 20. This will add a 20 second pause to your pipeline, which should give the Azure DevOps backend enough time to aggregate all coverage data before BQC reads it.