Qt wiki will be updated on October 12th 2023 starting at 11:30 AM (EEST) and the maintenance will last around 2-3 hours. During the maintenance the site will be unavailable.

CI Overview: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
No edit summary
(Updated Coin maintainer.)
 
(16 intermediate revisions by 7 users not shown)
Line 1: Line 1:
[[Category:Developing_Qt::QA]]
[[Category:Developing_Qt::QA]]
 
[[Category:Coin]]
[toc align_right="yes" depth="2"]


== Introduction ==
== Introduction ==
Line 9: Line 8:
The goals for the system are:
The goals for the system are:


* Implement an automated quality gate that blocks poor quality changes from entering the stable branch,
* Implement an automated quality gate that blocks poor quality changes from being merged in Git.
* Run an exhaustive set of configuration tests in a nightly fashion,
* Make test results available to developer in a clear, concise, and user friendly way.
* Make test results available to developer in a clear, concise, and user friendly way.


== Responsibilities ==
== Responsibilities ==


{| border="1"
{| class="wikitable"
! Task  
! Task  
! Task Description  
! Task Description  
Line 21: Line 19:
|-
|-
| General HW/VM maintenance  
| General HW/VM maintenance  
| Ensuring test equipment remains stable and working Ensuring we have the necessary hardware and software to meet Tier 1 testing needs
| Ensuring test equipment remains stable and working. Responsible for the architecture of the infrastructure.
| Tony Sarajärvi
| Tony Sarajärvi
|-
|-
| General scripting
| Coin
| Generic scripting work, i.e. the glue that makes the system work Maintenance of <tt>qtqa</tt> Qt5 module
| Make sure our Continuous Integration software (Coin) is up. Leads the development of the software.
| Tony Sarajärvi
| Toni Saario
|-
|-
| Open governance (Gerrit)  
| Open governance (Gerrit)  
| Scripting/maintenance relating to open test infrastructure (e.g. public build logs) Community point of contact for CI change requests
| Ensure that Gerrit's Git database is working and troubleshoot various error situations with it
| Tony Sarajärvi
|gerrit-admin@qt-project.org
|-
|-
| Support  
| Support  
| Front-line assistance with any issues arising from CI system  
| Front-line assistance with any issues arising from CI system  
| Tony Sarajärvi
| qt.ci@qt.io
|}
|}


== Continuous Integration (CI) System ==
== Continuous Integration (CI) System ==


All changes to Qt must be tested before integrated upstream:
All changes to Qt go through Gerrit (http://codereview.qt-project.org). A majority of projects there are tracked by a continuous integration system (Coin). A tracked project doesn't have a ''submit'' possibility for the commit, but instead has a ''merge patch for staging'' option. When a commit has been staged, it is judged by Coin whether or not it can be merged into upstream.
 
* All feature development must happen in a feature branch
* All bug fixing goes into Gerrit
** Continuously tested; check build and autotest results before integrating
 
Jenkins is used for continuous building and running autotests.


=== CI status ===
=== CI status ===
Line 51: Line 43:
*Patches currently being tested can be viewed with a Gerrit search: http://codereview.qt-project.org/#q,status:integrating,n,z
*Patches currently being tested can be viewed with a Gerrit search: http://codereview.qt-project.org/#q,status:integrating,n,z
*Patches staged for testing in the next build can also be viewed: http://codereview.qt-project.org/#q,status:staged,n,z
*Patches staged for testing in the next build can also be viewed: http://codereview.qt-project.org/#q,status:staged,n,z
*Coin's dashboard is visible here: https://testresults.qt.io/coin
*Current status of the infrastructure is found here: https://testresults.qt.io/grafana


=== Test Results ===
=== Test Results ===


Results are made available on http://testresults.qt.io/ci
All logs are published here https://testresults.qt.io/logs/, but knowing which log belongs to your build is quite difficult from that view. Instead one should see the links attached to the Gerrit commit or from Coin's dashboard to be able to find the right logs for your build.
 
At time of writing, results are made available at the end of each test run; a test run currently in progress cannot be viewed.


=== Qt5 Repository High Level Structure ===
=== Qt5 Repository High Level Structure ===
Line 66: Line 58:
The main components in the repository structure are:
The main components in the repository structure are:


'''Superproject repository'''
'''Superproject repository (qt/qt5.git)'''
'''''' This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.
 
'''''' '''Mainline superproject repository (e.g. qt/qt5.git)''':
* This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.
Only the Continuous Integration system pushes to this repository.
 
'''Module repositories'''
'''Module repositories'''
'''''' Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqta …)
 
* '''Mainline module repository (e.g. qt/qtbase.git)''':
* Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqa, ...)
Only the Continuous Integration system pushes to this repository.
 
'''Feature and other branches'''
'''Feature and other branches'''
'''''' Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.


== Why commits might get rejected by the CI System? ==
* Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.
 
====See also====
 
;How do I customize my test's execution?: If you need to change some aspect of your test's execution, such as the handling of the exit code, the test's maximum permitted runtime, or the commands used to execute the test, see [[CI Autotest Metadata]].
;What tests are we running ?:[[CI Configurations]]
;Reporting Bugs: [[Reporting Bugs#Reporting_bugs_for_Coin]]
: Bugs relating to the public autotest infrastructure should be reported in [https://bugreports.qt.io/ our bug-tracker] against the QTQAINFRA project.


For Qt 5 (mostly all the modules in [https://codereview.qt-project.org/ Gerrit]) all the autotests are supposed to compile and run successfully. So the actual procedure to reject a change is '''<span class="roughly"></span>''':
== What exactly does Coin do? ==
When Coin receives a notification that commits have been ''staged'' for a specific Qt module and a certain branch, it waits a while if further commits are being staged. This gives us the possibility to stage multiple commits in one go.


Coin goes through the following steps to determine if the commits are OK for merging:
* Compile the module and its dependencies
* Compile the module and its dependencies
* Run the autotest suite for the module under test (via `<code>make check</code>')
* Run the autotest suite for the module under test (via `<tt>make check</tt>')
* If the autotest suite succeed => the change gets '''integrated'''.
* If the autotest suite succeed => the change gets '''integrated'''.
* If one autotest fail => we run it again:
* If one autotest fail => we run it again:
** if it succeed, it's considered flaky => the change gets '''integrated''' anyway.
** if it succeed, it's considered flaky => the change gets '''integrated''' anyway.
** if it fails again, it's an stable regression => the change gets '''rejected'''.
** if it fails again, it's an stable regression => the change gets '''rejected'''.
The results are then published to http://testresults.qt.io and Gerrit is notified whether or not it should merge or reject the commit(s).
== Why commits might get rejected by the CI System? ==


Whenever a change gets rejected, you will get a new comment on Gerrit. The rejection might be caused by either a build failure or an autotests failure (regression).
For Qt 5 (mostly all the modules in [https://codereview.qt-project.org/ Gerrit]) all the autotests are supposed to compile and run successfully. The CI rejects the commit(s) for the following reasons:
* Sources don't compile
** Even though sources were tested locally by the developer, the CI also compiles the sources on multiple operating systems and different compilers.
* Autotests fail
** The commit(s) can either cause real regression, or the CI could have hit a flaky case somewhere else that's not related to the change at all. The CI unfortunately does not know this and rejects the commit(s).
* The CI infra fails
** This could be a bug in Coin, crashing hosts, network connection issues, time-outs etc. These problems are related to the stability of the CI. It's a continuous work as we try to improve the infrastructure so that we minimize the amount of rejects the infra itself causes.
Whenever a change gets rejected, you will get a new comment on Gerrit.


=== Reproducing an autotest failure ===
=== Reproducing an autotest failure ===
You can always try to reproduce the failure locally by just running ''make'' and ''make check'' under the ''tests/auto'' folder. However, understandably developers don't have every platform where the CI tests at their disposal. If encountering a situation like this, feel free to contact qt-ci@qt.io and we can discuss how to provide you with an environment where this can be reproduced.


You might want to try to reproduce what the CI System is doing (as there is a high hope that your changes are compiling and passing the autotests in your local environment).
For more information, see [[How to reproduce autotest fails]].
 
For this to be done, there are two relevant repositories containing the scripts that the CI System uses:
 
* http://qt.gitorious.org/qt/qtqa <- the test scripts themselves
* http://qt.gitorious.org/qtqa/testconfig <- the precise data for which scripts are used in which test configurations
 
Note: You might have to setup your perl dependencies for the <code>qtqa</code> module the very first time by reading:
 
<code>
# git clone git://gitorious.org/qt/qtqa.git _qtqa_latest
# perldoc _qtqa_latest/scripts/setup.pl
</code>
 
The scripts in qt/qtqa module may be used independently.
However, if you want to reproduce the same options and environment as used in the Qt CI system, you can invoke them through the test.pl script in the testconfig repository.
So, the basic steps are (for Linux):
 
<code>
# git clone git://gitorious.org/qt/qtbase.git
# cd qtbase
# git clone git://gitorious.org/qtqa/testconfig.git _testconfig
# perl _testconfig/test.pl —project 'QtBase master Integration' —stage 'linux-g++-32 Ubuntu 10.04 x86' 2>&amp;1 | tee testlog.txt
# <wait for the results&amp;amp;gt;
</code>
 
== How do I customize my test's execution? ==
 
If you need to change some aspect of your test's execution, such as the handling of the exit code, the test's maximum permitted runtime, or the commands used to execute the test, see [[CI_Autotest_Metadata]].
 
== What tests are we running ? ==
 
See [[CI_Configurations]].


== Exactly how are the CI machines configured? ==
== Exactly how are the CI machines configured? ==


See [[CI_Machine_Configuration]].
The virtual machines on which builds are done upon are set up by using scripts. These scripts are stored in Git, in qt5.git to be precise. In qt5.git repository there's a folder called ''/coin/platform_configurations'' that contains plain text files that list all the configuration that are run. The folder ''/coin/provisioning'' on the other hand contains all the scripts that are run to configure these VMs. The configuration scripts are publicly available and for anyone to modify and update.


== How can I see what the CI system is doing? ==
== How can I see what the CI system is doing? ==


The CI system's status for each project can be observed in near-realtime at http://testresults.qt.io/ci/status/
At the moment we don't have a real time view of what Coin is doing. However, as soon as a build is complete, its status is uploaded and [http://testresults.qt.io/coin the dashboard] is updated. The status page does not work if the CI system is down for maintenance or other reasons.
(or raw data at http://testresults.qt.io/cgi-bin/ci-api?pretty=true if you prefer). The status page does not work if the CI system is down for maintenance or other reasons.
 
Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt-project.org/pipermail/ci-reports/ . Mails contain various "X-Qt-CI-*" headers which may be useful for filtering.
 
== Reporting Bugs ==

Latest revision as of 07:32, 27 March 2020


Introduction

Shortening the time between the injection of a defect and detecting/fixing the problem has significant cost benefits. The QA team is maintaining a set of tools that provides a Continuous Integration (CI) infrastructure that enables SW development and QA to reach those goals.

The goals for the system are:

  • Implement an automated quality gate that blocks poor quality changes from being merged in Git.
  • Make test results available to developer in a clear, concise, and user friendly way.

Responsibilities

Task Task Description Point of Contact
General HW/VM maintenance Ensuring test equipment remains stable and working. Responsible for the architecture of the infrastructure. Tony Sarajärvi
Coin Make sure our Continuous Integration software (Coin) is up. Leads the development of the software. Toni Saario
Open governance (Gerrit) Ensure that Gerrit's Git database is working and troubleshoot various error situations with it gerrit-admin@qt-project.org
Support Front-line assistance with any issues arising from CI system qt.ci@qt.io

Continuous Integration (CI) System

All changes to Qt go through Gerrit (http://codereview.qt-project.org). A majority of projects there are tracked by a continuous integration system (Coin). A tracked project doesn't have a submit possibility for the commit, but instead has a merge patch for staging option. When a commit has been staged, it is judged by Coin whether or not it can be merged into upstream.

CI status

Test Results

All logs are published here https://testresults.qt.io/logs/, but knowing which log belongs to your build is quite difficult from that view. Instead one should see the links attached to the Gerrit commit or from Coin's dashboard to be able to find the right logs for your build.

Qt5 Repository High Level Structure

In Qt 5, various components of Qt have been split out into separate repositories, as part of the Qt modularization project.

This is intended to simplify the development of individual modules. Likewise, the repository and continuous integration setup with respect to individual modules is simpler, with only one level of continuous integration.

The main components in the repository structure are:

Superproject repository (qt/qt5.git)

  • This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.

Module repositories

  • Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqa, ...)

Feature and other branches

  • Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.

See also

How do I customize my test's execution?
If you need to change some aspect of your test's execution, such as the handling of the exit code, the test's maximum permitted runtime, or the commands used to execute the test, see CI Autotest Metadata.
What tests are we running ?
CI Configurations
Reporting Bugs
Reporting Bugs#Reporting_bugs_for_Coin
Bugs relating to the public autotest infrastructure should be reported in our bug-tracker against the QTQAINFRA project.

What exactly does Coin do?

When Coin receives a notification that commits have been staged for a specific Qt module and a certain branch, it waits a while if further commits are being staged. This gives us the possibility to stage multiple commits in one go.

Coin goes through the following steps to determine if the commits are OK for merging:

  • Compile the module and its dependencies
  • Run the autotest suite for the module under test (via `make check')
  • If the autotest suite succeed => the change gets integrated.
  • If one autotest fail => we run it again:
    • if it succeed, it's considered flaky => the change gets integrated anyway.
    • if it fails again, it's an stable regression => the change gets rejected.

The results are then published to http://testresults.qt.io and Gerrit is notified whether or not it should merge or reject the commit(s).

Why commits might get rejected by the CI System?

For Qt 5 (mostly all the modules in Gerrit) all the autotests are supposed to compile and run successfully. The CI rejects the commit(s) for the following reasons:

  • Sources don't compile
    • Even though sources were tested locally by the developer, the CI also compiles the sources on multiple operating systems and different compilers.
  • Autotests fail
    • The commit(s) can either cause real regression, or the CI could have hit a flaky case somewhere else that's not related to the change at all. The CI unfortunately does not know this and rejects the commit(s).
  • The CI infra fails
    • This could be a bug in Coin, crashing hosts, network connection issues, time-outs etc. These problems are related to the stability of the CI. It's a continuous work as we try to improve the infrastructure so that we minimize the amount of rejects the infra itself causes.

Whenever a change gets rejected, you will get a new comment on Gerrit.

Reproducing an autotest failure

You can always try to reproduce the failure locally by just running make and make check under the tests/auto folder. However, understandably developers don't have every platform where the CI tests at their disposal. If encountering a situation like this, feel free to contact qt-ci@qt.io and we can discuss how to provide you with an environment where this can be reproduced.

For more information, see How to reproduce autotest fails.

Exactly how are the CI machines configured?

The virtual machines on which builds are done upon are set up by using scripts. These scripts are stored in Git, in qt5.git to be precise. In qt5.git repository there's a folder called /coin/platform_configurations that contains plain text files that list all the configuration that are run. The folder /coin/provisioning on the other hand contains all the scripts that are run to configure these VMs. The configuration scripts are publicly available and for anyone to modify and update.

How can I see what the CI system is doing?

At the moment we don't have a real time view of what Coin is doing. However, as soon as a build is complete, its status is uploaded and the dashboard is updated. The status page does not work if the CI system is down for maintenance or other reasons.

Retrieved from "https://wiki-qt-io-staging.herokuapp.com/index.php?title=CI_Overview&oldid=36844"