QtMetrics How To Read

How to read autotest metrics from QtMetrics

The QtMetrics [testresults.qt.io] is a powerful tool, but can intimidate the everyday user as it presents you with a lot of numbers and statistics. This is a guide on how you find the relevant autotest data to you.

Below is the main view you receive when you enter the QtMetrics page. The first column shows different builds run. There are runs for all submodules, their master Qt5 (including all the submodules), and most if not all branches for them.

QtMetrics front page

On the left column listing the modules, select Qt5_release_Integration for example

Qt5_release emphasized

It will open a page like this. Now the left hand part is replaced with all the configurations run inside this job. Meaning, Qt5_release_Integration has been run on all those platforms and configurations.
Qt5_release front page

Going through these topics

Topics zoomed

• First column shows you if your build passed or not. This should be pretty clear. Abandoned means that it has been cancelled for some reason.
• The second column ‘forcesuccess’ tells you if the configuration has been flagged to forcefully pass even though there might have been a build break or a test failure.
• The third column is a bit less severe. ‘Insignificant’ means that tests are flagged as insignificant (at the CI level). This configuration needs to pass the compilation, but autotests may fail.

Neither forcesuccess or insignificant should be enabled. These are problems that need attention!

Next these ones

Numbers zoomed

These numbers simply show you how many tests have failed in this configuration. ‘Insignificant’ tests marked here are tests that are marked as insignificant in the .pro file of the test. It is not the CI system that has marked these as insignificant.

Let’s look closer at the autotests (further down the page).

Below the project view you have the list of all failing autotests. If we listed all autotests, the page would just bloat.
Autotest main

Now this part is important, and you really need to understand this part. Tests can be marked as insignificant at two levels:

• In the .pro file, usually marked by the developer
• In the CI configuration, usually marked by the CI team

If the test is marked insignificant in the .pro file, it will only mark the tests inside that tst_ -set as insignificant, and nothing else.
If the CI configuration is marked as insignificant, all the tests from that build configuration (e.g. linux-g++_static_Ubuntu_12.04_x64) are insignificant. It’s important to get rid of all these configurations, because no regression can be caught in that configuration as long as tests are marked as insignificant. Every time a new platform or configuration is introduced to the CI, it is initially marked as insignificant at this level so that it doesn’t block anything on the go and developers get time to fix these.

The same thing in QtMetrics. These two topics separate the significant and insignificant tests in the .pro -file level.
Autotest topics
Meaning, the left column shows how many times a test failed that the developer thought should be significant, alas we have a blocking problem here. The right column shows the tests that already the developer knows to be buggy, and are not relevant (There should be a bug report for all these. These tests are broken).

The colored columns
Autotest topics 2
explain the situation at the CI level. At start, every configuration is insignificant. But as tests pass once, the flag is removed, and tests start to be significant, and move to the blocking configuration.

The colors explain it too. The most fatal one is the darkest red. Whenever both the developer and the CI thinks, that a tests should pass, but didn’t, it fails the build and the commit doesn’t go through.

The second red column contains the situations where the developer thinks that his test is ok, but the CI is still marking that configuration as insignificant. These are usually remains that should be cleaned. Corrective action would be to go mark the test insignificant in the .pro file, so that the whole configuration could be marked as significant.

The orange columns contain the status of tests that have been marked insignificant already by the developer, as explained earlier. They are just split up accordingly to the CI configuration as well.

If I lost you now, I recommend you read the above a second time, or a third, and if that doesn’t help, contact me tosaraja [qt.io] for further help. That way I also know I need to explain this further.

How to find random failures, not appearing in the last build.
On the top of the page you have a time scale filter. Let’s move that back a couple of weeks
Time filter
Changed time filter

The update of the autotests below takes some time now. The QtMetrics page will go through a lot of XML files to parse for the requested data. But after it is done, you can notice how the numbers increased in this column. It will show you in total numbers how many times the test has failed and how many times it was run during that period. It will also show you a pass rate percentage.
Autotests increased
You can also find new test cases appearing. These are the really random ones I was referring to. That test didn’t fail during the last build, but it did fail once during that time frame we set in the time filter.
Autotests highlighted

By clicking on the test case, you can open further information about it.
Test case data
There you can see which test function is the failing one, which configuration it failed in, and which build it failed in.

I hope this page helps. If it still needs some more clarifications, please contact me tosaraja [qt.io]