Qt wiki will be updated on October 12th 2023 starting at 11:30 AM (EEST) and the maintenance will last around 2-3 hours. During the maintenance the site will be unavailable.

QtMetrics How To Read: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
No edit summary
(Add to HowTo)
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
h1. How to read autotest metrics from QtMetrics
[[Category:HowTo]]


The "QtMetrics":http://testresults.qt.io/qtmetrics/index.php is a powerful tool, but can intimidate the everyday user as it presents you with a lot of numbers and statistics. This is a guide on how you find the relevant autotest data to you.
= How to read autotest metrics from QtMetrics =
 
The [http://testresults.qt.io/qtmetrics/index.php QtMetrics] is a powerful tool, but can intimidate the everyday user as it presents you with a lot of numbers and statistics. This is a guide on how you find the relevant autotest data to you.


Below is the main view you receive when you enter the QtMetrics page. The first column shows different builds run. There are runs for all submodules, their master Qt5 (including all the submodules), and most if not all branches for them.
Below is the main view you receive when you enter the QtMetrics page. The first column shows different builds run. There are runs for all submodules, their master Qt5 (including all the submodules), and most if not all branches for them.


[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/1-metrics_front_page.png|QtMetrics front page]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/1-metrics_front_page.png


On the left column listing the modules, select Qt5_release_Integration for example
On the left column listing the modules, select Qt5_release_Integration for example


[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/2-qt5_release_emphasized.png|Qt5_release emphasized]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/2-qt5_release_emphasized.png


It will open a page like this. Now the left hand part is replaced with all the configurations run inside this job. Meaning, Qt5_release_Integration has been run on all those platforms and configurations.<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/3-qt5_release_front_page.png|Qt5_release front page]]
It will open a page like this. Now the left hand part is replaced with all the configurations run inside this job. Meaning, Qt5_release_Integration has been run on all those platforms and configurations.
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/3-qt5_release_front_page.png


Going through these topics
Going through these topics


[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/4-qt5_relese_topics_emphasized.png|Topics zoomed]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/4-qt5_relese_topics_emphasized.png


* First column shows you if your build passed or not. This should be pretty clear. Abandoned means that it has been cancelled for some reason.<br />* The second column 'forcesuccess' tells you if the configuration has been flagged to forcefully pass even though there might have been a build break or a test failure.<br />* The third column is a bit less severe. 'Insignificant' means that tests are flagged as insignificant (at the CI level). This configuration needs to pass the compilation, but autotests may fail.<br />'''Neither forcesuccess or insignificant should be enabled. These are problems that need attention!'''
* First column shows you if your build passed or not. This should be pretty clear. Abandoned means that it has been cancelled for some reason.
* The second column 'forcesuccess' tells you if the configuration has been flagged to forcefully pass even though there might have been a build break or a test failure.
* The third column is a bit less severe. 'Insignificant' means that tests are flagged as insignificant (at the CI level). This configuration needs to pass the compilation, but autotests may fail.
'''Neither forcesuccess or insignificant should be enabled. These are problems that need attention!'''


Next these ones
Next these ones


[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/5-qt5_relese_topics_emphasized2.png|Numbers zoomed]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/5-qt5_relese_topics_emphasized2.png


These numbers simply show you how many tests have failed in this configuration. 'Insignificant' tests marked here are tests that are marked as insignificant '''in the .pro file''' of the test. '''It is not the CI system that has marked these as insignificant'''.
These numbers simply show you how many tests have failed in this configuration. 'Insignificant' tests marked here are tests that are marked as insignificant '''in the .pro file''' of the test. '''It is not the CI system that has marked these as insignificant'''.
Line 27: Line 34:
Let's look closer at the autotests (further down the page).
Let's look closer at the autotests (further down the page).


Below the project view you have the list of all '''failing autotests'''. If we listed all autotests, the page would just bloat.<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/6-qt5_release_autotest_main.png|Autotest main]]
Below the project view you have the list of all '''failing autotests'''. If we listed all autotests, the page would just bloat.
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/6-qt5_release_autotest_main.png
 
Now this part is important, and you really need to understand this part. Tests can be marked as insignificant at two levels:
* In the .pro file, usually marked by the developer
* In the CI configuration, usually marked by the CI team
 
If the test is marked insignificant in the .pro file, it will only mark the tests inside that tst_ -set as insignificant, and nothing else.
If the CI configuration is marked as insignificant, all the tests from that build configuration (e.g. linux-g++_static_Ubuntu_12.04_x64) are insignificant. It's important to get rid of all these configurations, because no regression can be caught in that configuration as long as tests are marked as insignificant. Every time a new platform or configuration is introduced to the CI, it is initially marked as insignificant at this level so that it doesn't block anything on the go and developers get time to fix these.


Now this part is important, and you really need to understand this part. Tests can be marked as insignificant at two levels:<br />* In the .pro file, usually marked by the developer<br />* In the CI configuration, usually marked by the CI team
The same thing in QtMetrics. These two topics separate the significant and insignificant tests in the .pro -file level.


If the test is marked insignificant in the .pro file, it will only mark the tests inside that tst_ -set as insignificant, and nothing else.<br />If the CI configuration is marked as insignificant, all the tests from that build configuration (e.g. linux-g++_static_Ubuntu_12.04_x64) are insignificant. It's important to get rid of all these configurations, because no regression can be caught in that configuration as long as tests are marked as insignificant. Every time a new platform or configuration is introduced to the CI, it is initially marked as insignificant at this level so that it doesn't block anything on the go and developers get time to fix these.
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/7-qt5_release_autotest_main_topics.png


The same thing in QtMetrics. These two topics separate the significant and insignificant tests in the .pro -file level.<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/7-qt5_release_autotest_main_topics.png|Autotest topics]]<br />Meaning, the left column shows how many times a test failed that the developer thought should be significant, alas we have a blocking problem here. The right column shows the tests that already the developer knows to be buggy, and are not relevant (There should be a bug report for all these. These tests are broken).
Meaning, the left column shows how many times a test failed that the developer thought should be significant, alas we have a blocking problem here. The right column shows the tests that already the developer knows to be buggy, and are not relevant (There should be a bug report for all these. These tests are broken).


The colored columns<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/8-qt5_release_autotest_main_topics2.png|Autotest topics 2]]<br />explain the situation at the CI level. At start, every configuration is insignificant. But as tests pass once, the flag is removed, and tests start to be significant, and move to the blocking configuration.
The colored columns
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/8-qt5_release_autotest_main_topics2.png
explain the situation at the CI level. At start, every configuration is insignificant. But as tests pass once, the flag is removed, and tests start to be significant, and move to the blocking configuration.


The colors explain it too. The most fatal one is the darkest red. Whenever both the developer and the CI thinks, that a tests should pass, but didn't, it fails the build and the commit doesn't go through.
The colors explain it too. The most fatal one is the darkest red. Whenever both the developer and the CI thinks, that a tests should pass, but didn't, it fails the build and the commit doesn't go through.
Line 43: Line 61:
The orange columns contain the status of tests that have been marked insignificant already by the developer, as explained earlier. They are just split up accordingly to the CI configuration as well.
The orange columns contain the status of tests that have been marked insignificant already by the developer, as explained earlier. They are just split up accordingly to the CI configuration as well.


If I lost you now, I recommend you read the above a second time, or a third, and if that doesn't help, contact me "tosaraja":http://qt.io/mebmer/132527 for further help. That way I also know I need to explain this further.
If I lost you now, I recommend you read the above a second time, or a third, and if that doesn't help, contact me [http://qt.io/mebmer/132527 tosaraja] for further help. That way I also know I need to explain this further.
 
How to find random failures, not appearing in the last build.
On the top of the page you have a time scale filter. Let's move that back a couple of weeks
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/9-qt5_release_time_filter.png
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/10-qt5_release_time_filter_2.png
 
The update of the autotests below takes some time now. The QtMetrics page will go through a lot of XML files to parse for the requested data. But after it is done, you can notice how the numbers increased in this column. It will show you in total numbers how many times the test has failed and how many times it was run during that period. It will also show you a pass rate percentage.
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/11-qt5_release_autotests_more.png
 
You can also find new test cases appearing. These are the really random ones I was referring to. That test didn't fail during the last build, but it did fail once during that time frame we set in the time filter.
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/12-qt5_release_autotests_highlighted.png


How to find random failures, not appearing in the last build.<br />On the top of the page you have a time scale filter. Let's move that back a couple of weeks<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/9-qt5_release_time_filter.png|Time filter]]<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/10-qt5_release_time_filter_2.png|Changed time filter]]
By clicking on the test case, you can open further information about it.


The update of the autotests below takes some time now. The QtMetrics page will go through a lot of XML files to parse for the requested data. But after it is done, you can notice how the numbers increased in this column. It will show you in total numbers how many times the test has failed and how many times it was run during that period. It will also show you a pass rate percentage.<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/11-qt5_release_autotests_more.png|Autotests increased]]<br />You can also find new test cases appearing. These are the really random ones I was referring to. That test didn't fail during the last build, but it did fail once during that time frame we set in the time filter.<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/12-qt5_release_autotests_highlighted.png|Autotests highlighted]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/13qt5_release_test_case.png


By clicking on the test case, you can open further information about it.<br />[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/13qt5_release_test_case.png|Test case data]]<br />There you can see which test function is the failing one, which configuration it failed in, and which build it failed in.
There you can see which test function is the failing one, which configuration it failed in, and which build it failed in.


I hope this page helps. If it still needs some more clarifications, please contact me "tosaraja":http://qt.io/mebmer/132527
I hope this page helps. If it still needs some more clarifications, please contact me [http://qt.io/mebmer/132527 tosaraja]

Latest revision as of 16:07, 25 November 2016


How to read autotest metrics from QtMetrics

The QtMetrics is a powerful tool, but can intimidate the everyday user as it presents you with a lot of numbers and statistics. This is a guide on how you find the relevant autotest data to you.

Below is the main view you receive when you enter the QtMetrics page. The first column shows different builds run. There are runs for all submodules, their master Qt5 (including all the submodules), and most if not all branches for them.

1-metrics_front_page.png

On the left column listing the modules, select Qt5_release_Integration for example

2-qt5_release_emphasized.png

It will open a page like this. Now the left hand part is replaced with all the configurations run inside this job. Meaning, Qt5_release_Integration has been run on all those platforms and configurations.

3-qt5_release_front_page.png

Going through these topics

4-qt5_relese_topics_emphasized.png

  • First column shows you if your build passed or not. This should be pretty clear. Abandoned means that it has been cancelled for some reason.
  • The second column 'forcesuccess' tells you if the configuration has been flagged to forcefully pass even though there might have been a build break or a test failure.
  • The third column is a bit less severe. 'Insignificant' means that tests are flagged as insignificant (at the CI level). This configuration needs to pass the compilation, but autotests may fail.

Neither forcesuccess or insignificant should be enabled. These are problems that need attention!

Next these ones

5-qt5_relese_topics_emphasized2.png

These numbers simply show you how many tests have failed in this configuration. 'Insignificant' tests marked here are tests that are marked as insignificant in the .pro file of the test. It is not the CI system that has marked these as insignificant.

Let's look closer at the autotests (further down the page).

Below the project view you have the list of all failing autotests. If we listed all autotests, the page would just bloat.

6-qt5_release_autotest_main.png

Now this part is important, and you really need to understand this part. Tests can be marked as insignificant at two levels:

  • In the .pro file, usually marked by the developer
  • In the CI configuration, usually marked by the CI team

If the test is marked insignificant in the .pro file, it will only mark the tests inside that tst_ -set as insignificant, and nothing else. If the CI configuration is marked as insignificant, all the tests from that build configuration (e.g. linux-g++_static_Ubuntu_12.04_x64) are insignificant. It's important to get rid of all these configurations, because no regression can be caught in that configuration as long as tests are marked as insignificant. Every time a new platform or configuration is introduced to the CI, it is initially marked as insignificant at this level so that it doesn't block anything on the go and developers get time to fix these.

The same thing in QtMetrics. These two topics separate the significant and insignificant tests in the .pro -file level.

7-qt5_release_autotest_main_topics.png

Meaning, the left column shows how many times a test failed that the developer thought should be significant, alas we have a blocking problem here. The right column shows the tests that already the developer knows to be buggy, and are not relevant (There should be a bug report for all these. These tests are broken).

The colored columns 8-qt5_release_autotest_main_topics2.png explain the situation at the CI level. At start, every configuration is insignificant. But as tests pass once, the flag is removed, and tests start to be significant, and move to the blocking configuration.

The colors explain it too. The most fatal one is the darkest red. Whenever both the developer and the CI thinks, that a tests should pass, but didn't, it fails the build and the commit doesn't go through.

The second red column contains the situations where the developer thinks that his test is ok, but the CI is still marking that configuration as insignificant. These are usually remains that should be cleaned. Corrective action would be to go mark the test insignificant in the .pro file, so that the whole configuration could be marked as significant.

The orange columns contain the status of tests that have been marked insignificant already by the developer, as explained earlier. They are just split up accordingly to the CI configuration as well.

If I lost you now, I recommend you read the above a second time, or a third, and if that doesn't help, contact me tosaraja for further help. That way I also know I need to explain this further.

How to find random failures, not appearing in the last build. On the top of the page you have a time scale filter. Let's move that back a couple of weeks 9-qt5_release_time_filter.png 10-qt5_release_time_filter_2.png

The update of the autotests below takes some time now. The QtMetrics page will go through a lot of XML files to parse for the requested data. But after it is done, you can notice how the numbers increased in this column. It will show you in total numbers how many times the test has failed and how many times it was run during that period. It will also show you a pass rate percentage.

11-qt5_release_autotests_more.png

You can also find new test cases appearing. These are the really random ones I was referring to. That test didn't fail during the last build, but it did fail once during that time frame we set in the time filter.

12-qt5_release_autotests_highlighted.png

By clicking on the test case, you can open further information about it.

13qt5_release_test_case.png

There you can see which test function is the failing one, which configuration it failed in, and which build it failed in.

I hope this page helps. If it still needs some more clarifications, please contact me tosaraja