CI Overview

Introduction

Shortening the time between the injection of a defect and detecting/fixing the problem has significant cost benefits. The QA team is maintaining a set of tools that provides a Continuous Integration (CI) infrastructure that enables SW development and QA to reach those goals.

The goals for the system are:

• Implement an automated quality gate that blocks poor quality changes from entering the stable branch,
• Run an exhaustive set of configuration tests in a nightly fashion,
• Make test results available to developer in a clear, concise, and user friendly way.

Responsibilities

Task	Task Description	Point of Contact
General HW/VM maintenance	Ensuring test equipment remains stable and working Ensuring we have the necessary hardware and software to meet Tier 1 testing needs	Tony Sarajärvi
General scripting	Generic scripting work, i.e. the glue that makes the system work Maintenance of qtqa Qt5 module	Tony Sarajärvi
Open governance (Gerrit)	Scripting/maintenance relating to open test infrastructure (e.g. public build logs) Community point of contact for CI change requests	Tony Sarajärvi
Support	Front-line assistance with any issues arising from CI system	Tony Sarajärvi

Continuous Integration (CI) System

All changes to Qt must be tested before integrated upstream:

• All feature development must happen in a feature branch
• All bug fixing goes into Gerrit
  • Continuously tested; check build and autotest results before integrating

Jenkins is used for continuous building and running autotests.

CI status

• Patches currently being tested can be viewed with a Gerrit search: http://codereview.qt-project.org/#q,status:integrating,n,z
• Patches staged for testing in the next build can also be viewed: http://codereview.qt-project.org/#q,status:staged,n,z

Test Results

Results are made available on http://testresults.qt.io/ci

At time of writing, results are made available at the end of each test run; a test run currently in progress cannot be viewed.

Qt5 Repository High Level Structure

In Qt 5, various components of Qt have been split out into separate repositories, as part of the Qt modularization project.

This is intended to simplify the development of individual modules. Likewise, the repository and continuous integration setup with respect to individual modules is simpler, with only one level of continuous integration.

The main components in the repository structure are:

Superproject repository

• This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.
• Mainline superproject repository (e.g. qt/qt5.git): Only the Continuous Integration system pushes to this repository.

Module repositories

• Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqta, ...)
• Mainline module repository (e.g. qt/qtbase.git): Only the Continuous Integration system pushes to this repository.

Feature and other branches

• Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.

Why commits might get rejected by the CI System?

For Qt 5 (mostly all the modules in Gerrit) all the autotests are supposed to compile and run successfully. So the actual procedure to reject a change is roughly:

• Compile the module and its dependencies
• Run the autotest suite for the module under test (via `make check')
• If the autotest suite succeed => the change gets integrated.
• If one autotest fail => we run it again:
  • if it succeed, it's considered flaky => the change gets integrated anyway.
  • if it fails again, it's an stable regression => the change gets rejected.

Whenever a change gets rejected, you will get a new comment on Gerrit. The rejection might be caused by either a build failure or an autotests failure (regression).

Reproducing an autotest failure

You might want to try to reproduce what the CI System is doing (as there is a high hope that your changes are compiling and passing the autotests in your local environment).

For this to be done, there are two relevant repositories containing the scripts that the CI System uses:

• http://qt.gitorious.org/qt/qtqa <- the test scripts themselves
• http://qt.gitorious.org/qtqa/testconfig <- the precise data for which scripts are used in which test configurations

Note: You might have to setup your perl dependencies for the qtqa module the very first time by reading:

# git clone git://gitorious.org/qt/qtqa.git _qtqa_latest
# perldoc _qtqa_latest/scripts/setup.pl

The scripts in qt/qtqa module may be used independently. However, if you want to reproduce the same options and environment as used in the Qt CI system, you can invoke them through the test.pl script in the testconfig repository. So, the basic steps are (for Linux):

# git clone git://gitorious.org/qt/qtbase.git
# cd qtbase
# git clone git://gitorious.org/qtqa/testconfig.git _testconfig
# perl _testconfig/test.pl —project 'QtBase master Integration' —stage 'linux-g++-32 Ubuntu 10.04 x86' 2>&1 | tee testlog.txt
# <wait for the results&amp;amp;gt;

How do I customize my test's execution?

If you need to change some aspect of your test's execution, such as the handling of the exit code, the test's maximum permitted runtime, or the commands used to execute the test, see CI Autotest Metadata.

What tests are we running ?

See CI Configurations.

Exactly how are the CI machines configured?

See CI Machine Configuration.

How can I see what the CI system is doing?

The CI system's status for each project can be observed in near-realtime at http://testresults.qt.io/ci/status/ (or raw data at http://testresults.qt.io/cgi-bin/ci-api?pretty=true if you prefer). The status page does not work if the CI system is down for maintenance or other reasons.

Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt-project.org/pipermail/ci-reports/ . Mails contain various "X-Qt-CI-*" headers which may be useful for filtering.

Reporting Bugs

Bugs relating to the public autotest infrastructure should be reported on https://bugreports.qt.io/ against the QTQAINFRA project.