Qt wiki will be updated on October 12th 2023 starting at 11:30 AM (EEST) and the maintenance will last around 2-3 hours. During the maintenance the site will be unavailable.
CI Overview: Difference between revisions
mNo edit summary |
(Updated page to represent 2018 status.) |
||
| Line 7: | Line 7: | ||
The goals for the system are: | The goals for the system are: | ||
* Implement an automated quality gate that blocks poor quality changes from | * Implement an automated quality gate that blocks poor quality changes from being merged in Git. | ||
* Make test results available to developer in a clear, concise, and user friendly way. | * Make test results available to developer in a clear, concise, and user friendly way. | ||
| Line 19: | Line 18: | ||
|- | |- | ||
| General HW/VM maintenance | | General HW/VM maintenance | ||
| Ensuring test equipment remains stable and working | | Ensuring test equipment remains stable and working. Responsible for the architecture of the infrastructure. | ||
| Tony Sarajärvi | | Tony Sarajärvi | ||
|- | |- | ||
| | | Coin | ||
| | | Make sure our Continuous Integration software (Coin) is up. Leads the development of the software. | ||
| | | Aapo Keskimölö | ||
|- | |- | ||
| Open governance (Gerrit) | | Open governance (Gerrit) | ||
| | | Ensure that Gerrit's Git database is working and troubleshoot various error situations with it | ||
| | | Oswald Buddenhagen | ||
|- | |- | ||
| Support | | Support | ||
| Front-line assistance with any issues arising from CI system | | Front-line assistance with any issues arising from CI system | ||
| | | qt.ci@qt.io | ||
|} | |} | ||
== Continuous Integration (CI) System == | == Continuous Integration (CI) System == | ||
All changes to Qt | All changes to Qt go through Gerrit (http://codereview.qt-project.org). A majority of projects there are tracked by a continuous integration system (Coin). A tracked project doesn't have a ''submit'' possibility for the commit, but instead has a ''merge patch for staging'' option. When a commit has been staged, it is judged by Coin whether or not it can be merged into upstream. | ||
=== CI status === | === CI status === | ||
| Line 49: | Line 42: | ||
*Patches currently being tested can be viewed with a Gerrit search: http://codereview.qt-project.org/#q,status:integrating,n,z | *Patches currently being tested can be viewed with a Gerrit search: http://codereview.qt-project.org/#q,status:integrating,n,z | ||
*Patches staged for testing in the next build can also be viewed: http://codereview.qt-project.org/#q,status:staged,n,z | *Patches staged for testing in the next build can also be viewed: http://codereview.qt-project.org/#q,status:staged,n,z | ||
*Coin's dashboard is visible here: https://testresults.qt.io/coin | |||
*Current status of the infrastructure is found here: https://testresults.qt.io/grafana | |||
=== Test Results === | === Test Results === | ||
All logs are published here https://testresults.qt.io/logs/, but knowing which log belongs to your build is quite difficult from that view. Instead one should see the links attached to the Gerrit commit or from Coin's dashboard to be able to find the right logs for your build. | |||
=== Qt5 Repository High Level Structure === | === Qt5 Repository High Level Structure === | ||
| Line 64: | Line 57: | ||
The main components in the repository structure are: | The main components in the repository structure are: | ||
'''Superproject repository''' | '''Superproject repository (qt/qt5.git)''' | ||
* This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt. | * This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt. | ||
'''Module repositories''' | '''Module repositories''' | ||
* Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, | * Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqa, ...) | ||
'''Feature and other branches''' | '''Feature and other branches''' | ||
| Line 78: | Line 69: | ||
* Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens. | * Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens. | ||
== | == What exactly does Coin do? == | ||
When Coin receives a notification that commits have been ''staged'' for a specific Qt module and a certain branch, it waits a while if further commits are being staged. This gives us the possibility to stage multiple commits in one go. | |||
Coin goes through the following steps to determine if the commits are OK for merging: | |||
* Compile the module and its dependencies | * Compile the module and its dependencies | ||
* Run the autotest suite for the module under test (via `<tt>make check</tt>') | * Run the autotest suite for the module under test (via `<tt>make check</tt>') | ||
| Line 88: | Line 79: | ||
** if it succeed, it's considered flaky => the change gets '''integrated''' anyway. | ** if it succeed, it's considered flaky => the change gets '''integrated''' anyway. | ||
** if it fails again, it's an stable regression => the change gets '''rejected'''. | ** if it fails again, it's an stable regression => the change gets '''rejected'''. | ||
The results are then published to http://testresults.qt.io and Gerrit is notified whether or not it should merge or reject the commit(s). | |||
Whenever a change gets rejected, you will get a new comment on Gerrit | == Why commits might get rejected by the CI System? == | ||
For Qt 5 (mostly all the modules in [https://codereview.qt-project.org/ Gerrit]) all the autotests are supposed to compile and run successfully. The CI rejects the commit(s) for the following reasons: | |||
* Sources don't compile | |||
** Even though sources were tested locally by the developer, the CI also compiles the sources on multiple operating systems and different compilers. | |||
* Autotests fail | |||
** The commit(s) can either cause real regression, or the CI could have hit a flaky case somewhere else that's not related to the change at all. The CI unfortunately does not know this and rejects the commit(s). | |||
* The CI infra fails | |||
** This could be a bug in Coin, crashing hosts, network connection issues, time-outs etc. These problems are related to the stability of the CI. It's a continuous work as we try to improve the infrastructure so that we minimize the amount of rejects the infra itself causes. | |||
Whenever a change gets rejected, you will get a new comment on Gerrit. | |||
=== Reproducing an autotest failure === | === Reproducing an autotest failure === | ||
You can always try to reproduce the failure locally by just running ''make'' and ''make check'' under the ''tests/auto'' folder. However, understandably developers don't have every platform where the CI tests at their disposal. If encountering a situation like this, feel free to contact qt-ci@qt.io and we can discuss how to provide you with an environment where this can be reproduced. | |||
You | |||
== How do I customize my test's execution? == | == How do I customize my test's execution? == | ||
| Line 129: | Line 105: | ||
== Exactly how are the CI machines configured? == | == Exactly how are the CI machines configured? == | ||
The virtual machines on which builds are done upon are set up by using scripts. These scripts are stored in Git, in qt5.git to be precise. In qt5.git repository there's a folder called ''/coin/platform_configurations'' that contains plain text files that list all the configuration that are run. The folder ''/coin/provisioning'' on the other hand contains all the scripts that are run to configure these VMs. The configuration scripts are publicly available and for anyone to modify and update. | |||
== How can I see what the CI system is doing? == | == How can I see what the CI system is doing? == | ||
At the moment we don't have a real time view of what Coin is doing. However, as soon as a build is complete, its status is uploaded and dashboard at http://testresults.qt.io/coin is updated. The status page does not work if the CI system is down for maintenance or other reasons. | |||
Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt-project.org/pipermail/ci-reports/ . Mails contain various "X-Qt-CI-*" headers which may be useful for filtering. | Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt-project.org/pipermail/ci-reports/ . Mails contain various "X-Qt-CI-*" headers which may be useful for filtering. | ||
Revision as of 09:17, 12 June 2018
Introduction
Shortening the time between the injection of a defect and detecting/fixing the problem has significant cost benefits. The QA team is maintaining a set of tools that provides a Continuous Integration (CI) infrastructure that enables SW development and QA to reach those goals.
The goals for the system are:
- Implement an automated quality gate that blocks poor quality changes from being merged in Git.
- Make test results available to developer in a clear, concise, and user friendly way.
Responsibilities
| Task | Task Description | Point of Contact |
|---|---|---|
| General HW/VM maintenance | Ensuring test equipment remains stable and working. Responsible for the architecture of the infrastructure. | Tony Sarajärvi |
| Coin | Make sure our Continuous Integration software (Coin) is up. Leads the development of the software. | Aapo Keskimölö |
| Open governance (Gerrit) | Ensure that Gerrit's Git database is working and troubleshoot various error situations with it | Oswald Buddenhagen |
| Support | Front-line assistance with any issues arising from CI system | qt.ci@qt.io |
Continuous Integration (CI) System
All changes to Qt go through Gerrit (http://codereview.qt-project.org). A majority of projects there are tracked by a continuous integration system (Coin). A tracked project doesn't have a submit possibility for the commit, but instead has a merge patch for staging option. When a commit has been staged, it is judged by Coin whether or not it can be merged into upstream.
CI status
- Patches currently being tested can be viewed with a Gerrit search: http://codereview.qt-project.org/#q,status:integrating,n,z
- Patches staged for testing in the next build can also be viewed: http://codereview.qt-project.org/#q,status:staged,n,z
- Coin's dashboard is visible here: https://testresults.qt.io/coin
- Current status of the infrastructure is found here: https://testresults.qt.io/grafana
Test Results
All logs are published here https://testresults.qt.io/logs/, but knowing which log belongs to your build is quite difficult from that view. Instead one should see the links attached to the Gerrit commit or from Coin's dashboard to be able to find the right logs for your build.
Qt5 Repository High Level Structure
In Qt 5, various components of Qt have been split out into separate repositories, as part of the Qt modularization project.
This is intended to simplify the development of individual modules. Likewise, the repository and continuous integration setup with respect to individual modules is simpler, with only one level of continuous integration.
The main components in the repository structure are:
Superproject repository (qt/qt5.git)
- This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.
Module repositories
- Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqa, ...)
Feature and other branches
- Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.
What exactly does Coin do?
When Coin receives a notification that commits have been staged for a specific Qt module and a certain branch, it waits a while if further commits are being staged. This gives us the possibility to stage multiple commits in one go.
Coin goes through the following steps to determine if the commits are OK for merging:
- Compile the module and its dependencies
- Run the autotest suite for the module under test (via `make check')
- If the autotest suite succeed => the change gets integrated.
- If one autotest fail => we run it again:
- if it succeed, it's considered flaky => the change gets integrated anyway.
- if it fails again, it's an stable regression => the change gets rejected.
The results are then published to http://testresults.qt.io and Gerrit is notified whether or not it should merge or reject the commit(s).
Why commits might get rejected by the CI System?
For Qt 5 (mostly all the modules in Gerrit) all the autotests are supposed to compile and run successfully. The CI rejects the commit(s) for the following reasons:
- Sources don't compile
- Even though sources were tested locally by the developer, the CI also compiles the sources on multiple operating systems and different compilers.
- Autotests fail
- The commit(s) can either cause real regression, or the CI could have hit a flaky case somewhere else that's not related to the change at all. The CI unfortunately does not know this and rejects the commit(s).
- The CI infra fails
- This could be a bug in Coin, crashing hosts, network connection issues, time-outs etc. These problems are related to the stability of the CI. It's a continuous work as we try to improve the infrastructure so that we minimize the amount of rejects the infra itself causes.
Whenever a change gets rejected, you will get a new comment on Gerrit.
Reproducing an autotest failure
You can always try to reproduce the failure locally by just running make and make check under the tests/auto folder. However, understandably developers don't have every platform where the CI tests at their disposal. If encountering a situation like this, feel free to contact qt-ci@qt.io and we can discuss how to provide you with an environment where this can be reproduced.
How do I customize my test's execution?
If you need to change some aspect of your test's execution, such as the handling of the exit code, the test's maximum permitted runtime, or the commands used to execute the test, see CI Autotest Metadata.
What tests are we running ?
See CI Configurations.
Exactly how are the CI machines configured?
The virtual machines on which builds are done upon are set up by using scripts. These scripts are stored in Git, in qt5.git to be precise. In qt5.git repository there's a folder called /coin/platform_configurations that contains plain text files that list all the configuration that are run. The folder /coin/provisioning on the other hand contains all the scripts that are run to configure these VMs. The configuration scripts are publicly available and for anyone to modify and update.
How can I see what the CI system is doing?
At the moment we don't have a real time view of what Coin is doing. However, as soon as a build is complete, its status is uploaded and dashboard at http://testresults.qt.io/coin is updated. The status page does not work if the CI system is down for maintenance or other reasons.
Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt-project.org/pipermail/ci-reports/ . Mails contain various "X-Qt-CI-*" headers which may be useful for filtering.
Reporting Bugs
Bugs relating to the public autotest infrastructure should be reported on https://bugreports.qt.io/ against the QTQAINFRA project.