Qt wiki will be updated on October 12th 2023 starting at 11:30 AM (EEST) and the maintenance will last around 2-3 hours. During the maintenance the site will be unavailable.
Qt Metrics 2 Description
| UNDER CONSTRUCTION |
INTRODUCTION
The Qt Metrics 2 web portal visualizes the progress of Qt's quality, focusing on Continuous Integration (CI). The goal is to automate laborious manual work that is required, for example, in the CI release cycle, to report the key information, as well as providing real time data with good performance. The Qt integration teams (in The Qt Company) and the global Qt developer community are the target audience.
Qt Metrics 2: http://testresults.qt.io/qtmetrics/
Version: This document is updated against the report builder v1.0
This document is divided in chapters; each chapter having its own purpose and target audience:
2. Using the System: What (what the users see)
3. Detailed Description: How it works (quick overview to implementation)
4. Developing and Maintenance: How to develop it (how to create new metrics/pages)
System Overview
The Qt Metrics system consists of three main components: the parser, the report builder and the database. See the picture below. The parser reads the build logs, test results, or other files that are created in the development and integration process, and pushes the data into the database. The report builder reads the data from the database, and shows predefined metrics and reports based on user's area of interest. The report builder is implemented with PHP (Slim framework), Bootstrap, HTML, CSS, and JavaScript with some jQuery and D3 graphics components. The parser is implemented with Perl (implementation for the 'new-CI' not done yet and solution therefore still open). The database is a MariaDB database.
PICTURE. Generic system architecture
Requirements and Target Audience
System Requirements
The following general requirements have been identified:
- Because Qt is based on open source, it is recommended to use open source technologies on both the metrics system backend and the frontend solutions. The selected technologies and technical solutions shall fit also for other areas (in addition to CI) that will be implemented later on.
- Anyone in the Qt developer community should be able to create new reports on the metrics pages.
- The user of the web site should be able to filter the information based on different criteria and to browse relevant information for his or her needs.
User stories
The target audience for the metrics site are the Qt integration teams (in The Qt Company) and the global Qt developer community.
The following user needs have been identified (a need may interest several roles although only one listed):
- As a Qt developer I want to see have autotests on my area failed recently (e.g. tst_qdbusconnection)
- As a Qt developer I want to know about a Qt module/project (e.g. qtbase) and see why it fails so often
- As a Qt developer I want to see which test I should fix to improve our coverage (e.g. I look at the module and see that tst_qwidget is red on most platforms, I click a link and dive deeper and how see how this test fails)
- As a module integrator I want to see the status of a Qt module/project (e.g. qtdeclarative)
- As a platform integrator I want to see the status of my platform (e.g. Windows), see which tests fail often and which ones are insignificant on this platform, and to click a link to a test to see more details.
- As a platform integrator I want to easily study a recent build log
- As a platform integrator I want to see if a configuration is untested because it's insignificant
- As a platform integrator I want to see if a test function or row is untested because it's blacklisted, and whether the blacklisting could be removed because the blacklisted test has passed in recent builds
- As a platform integrator I want to see a report of the most statistically relevant failing/flaky tests of the past week in order to start fixing the most relevant things step by step and see the status to improve
- As a tech lead I want to be able to dive from overview level into detailed data
- As a tech lead I want to see statistics
- As a release manager I want to see the status of a forthcoming release (e.g. 5.5.1)
- As a CI support person I want to see which test failed lately (randomly)
- As a CI support person I want to see which tests could be raised from insignificant to significant
- As a CI support person I want to be able control the data shown on the site, e.g. the old branches could be hidden or removed, and data older than for example three months could be removed
- As a manager I want to see an easy-to-read overview of the CI status
See also the ch. FUTURE DEVELOPMENT.
USING THE SYSTEM
This section introduces the Qt Metrics system from the user point of view.
Data Structure
The data is structured now so that only the (daily) Qt5 state builds are included. You can view either the Qt5 state build results, or the test results in the Qt5 state builds, across the branches.
The hierarchy of the build data is as follows:
- project ("Qt5"), state ("state") and branch (e.g. "dev")
- configuration (e.g. "macx-clang_developer-build_OSX_10.10")
The hierarchy of the test data, run in a build, is as follows:
- project (e.g. "QtBase")
- testset (e.g. "tst_qftp")
- test function (e.g. "connectToHost")
- test row (e.g. "WithoutProxy:ok01")
Functionality
With the Qt Metrics you can for example:
- see a quick overview to the latest build and test status (as a dasboard or a data view)
- see the build status for Windows platform configurations
- see test status for all testsets in a project, for example "QtBase"
- see which testsets, test functions and test rows are failing in a configuration, for example "macx-clang_developer-build_OSX_10.10"
- see top failing testsets and test functions
- see the list of flaky testsets
- see the top duration list of testsets (testsets that take the most time to execute)
- see the blacklisted test functions and test rows that are currently passing in recent builds, and therefore the blacklisted tag could maybe be removed
The site map page provides information on moving between the pages.
The "i" button on every page provides detailed information of the views and notation used.
Note that (in most views) the passed test results are excluded and only the failed or skipped (or blacklisted) results are shown. This is to keep the data amount at minimum and the views easily readable.
Indicators
A striped animation bar (either in the header row or in the main area) is used to indicate that the data is being retrieved from the database. Most pages should open in few seconds but some of the list views may take a bit longer to load.
The green "O" indicator in the header row indicates the status of build data, either the time of last data update or the status of ongoing build log parsing. Move mouse over the icon to see the status, or press it to update the page when also the icon and the status indicated gets updated.
Database Administration
Certain database operations can be done via an administration interface, please contact the CI team or anyone from the support team for access. You can check the size of the database tables, number of builds and last build time per branch, and number of daily builds per state.
With the following operations the database size, and the overall performance of the database and the web UI can kept at optimum level:
- archive a branch to hide it from the views without deleting the data itself from the database
- remove a branch to permanently delete the build and test data related to the branch from the database
- remove all the build and test data (across all branches) for a selected day (for selected state)
Note: If an old branch needs to be removed, it is recommended to first remove old data for selected days, and then remove the branch. This is because removing one (project_run) build takes a few seconds to delete the data from the full table hierarchy. Removing a branch with hundreds of (project_run) builds will lock the tables resulting to the situation where the web UI as well as the operations to parse and save a new build might be blocked for a long time.
DETAILED DESCRIPTION
This section discusses the Qt Metrics system components and their implementation into a certain level of detail. The target audience is the SW designer(s) maintaining and developing the system.
Report Builder
The report builder is implemented with PHP (Slim framework), Bootstrap, HTML, CSS, and JavaScript with some jQuery and D3 graphics components.
PHP and Slim framework
PHP (http://php.net/) is a popular general-purpose scripting language that is especially suited to web development. Slim (http://www.slimframework.com/) is a PHP micro framework that helps writing simple yet powerful web applications and APIs. The Slim Framework helps to map resource URIs to callback functions for specific HTTP request methods (e.g. GET, POST, PUT, DELETE). A Slim application will invoke the first route that matches the current HTTP request’s URI and method.
The Slim supports the MVC (Model-View-Controller) design pattern, implemented in the Qt Metrics with the database classes and UI (HTML) templates. The routing of URIs with models and views is implemented in the index.php file.
Twig and Slim Views
Twig (http://twig.sensiolabs.org/) is a server-side template engine for PHP. It processes and combines provided dynamic data with static HTML content produce the final web document for the browser. Slim Views (https://github.com/slimphp/Slim-Views) contains custom view classes for the Twig template framework.
The content of the Qt Metrics pages are produced with Twig in the Slim template HTML files, combining the data from the routing system. Twig is used to loop the input data and for example to dynamically select the icons based on build or test results.
Bootstrap
Bootstrap (http://getbootstrap.com/) is the most popular HTML, CSS, and JS framework for developing responsive, mobile first projects on the web. Bootstrap is compatible with the latest versions of the Google Chrome, Firefox, Internet Explorer, Opera, and Safari browsers. Bootstrap requires jQuery to be included for its JavaScript components.
The Qt Metrics utilizes for example the grid system, tables, buttons, glyphicons, navbars, breadcrumbs, labels, badges, jumbotron, alerts, progress bars, wells, modals and tooltips.
jQuery and jQuery UI
jQuery (https://jquery.com/) is jQuery is a cross-platform JavaScript library designed to simplify things the client-side things like HTML scripting, event handling, animation, and Ajax. jQuery UI (https://jqueryui.com/) is a curated set of user interface interactions, effects, widgets, and themes built on top of the jQuery JavaScript library.
The Qt Metrics utilizes for example jQuery Ajax and jQuery UI Autocomplete.
D3
D3 (Data-Driven Documents; http://d3js.org/) is a JavaScript library for manipulating documents based on data, and helps bringing data to life using HTML, SVG and CSS. The http://d3js.org/ and https://github.com/mbostock/d3/wiki/Gallery include guidance and examples, and because of the popularity of D3 many detailed examples are available in the Internet.
The Qt Metrics utilizes D3 for example for the meters on the dashboard page.
Parser
CI build data is added to the database only by the parser. There is not any need to update the data for a single build afterwards. The Qt Metrics UI provides means to archive or remove the data from the database.
'Old-CI' parser
The data is inserted from the build and test result files when a new build has completed. This method is called a "single scan". The data already in the database is never updated since build results never change after a build. However, if there is a need for example to add a new database field or to change a field type, the whole database must be rebuilt from the log files. This method is referred as a "full scan". There is also a "reload" method to manually delete and re-create the data for a specified build. In addition, there several options what data is printed when running the parser (use "help" option to list all).
The parser is implemented with Perl. The parser first reads the build log file and the (possibly existing) test result XML files into a hash. The parser uses Archive::Zip and XML::LibXML to open and read the XML archives and files. Then the database is checked whether the required tables and indexes exist, and creates them if necessary. Finally the hash data is inserted into the database top-down in the database table hierarchy. This is because upper level data (e.g. conf_run.id) is needed in order to refer to them in the lower level tables (e.g. testset_run.conf_run_id).
It must be noted that the test data for a build may still be missing after the parser is run. This is for example because there are not any test result XML files available, the test result XML files are not valid for the XML reader (e.g. XML tags missing or such characters used that are not supported in the specified character set), or duplicate tag names are used for test functions or test rows (in such a case only the last one will get saved).
'New-CI' parser
The parser for the new-CI is yet to be made...
Database
The database is a MariaDB database, and all tables use MyISAM storage engine. The database is running on the Qt testresults server on the qt.io domain.
PICTURE. Database tables, their fields, and field types
Data is parsed from the build log files by default, or from the XML test reports if separately mentioned so in the tables below.
| Table | Purpose |
|---|---|
| project | Project names, both build projects (currently only the 'Qt5' included) and the parent projects for the testsets (like 'qtbase') |
| branch | Branch name (like '5.5' or 'dev') |
| state | State name (currently only the 'state' builds are included) |
| platform | Target or host platform (like 'linux' / Ubuntu_11.10' / 'x86') |
| compiler | Target or host compiler (like 'g++') |
| conf | Build configuration consisting of target and host platform and compiler, features and full name (like 'linux-g++_shadow-build_Ubuntu_11.10_x86') |
| phase | Phase of the configuration build (like 'configuring Qt') |
| testset | Testset for its parent project (like 'tst_qftp') |
| testfunction | Testfunction for its parent testset (like 'initTestCase'), parsed from XML reports:
<TestFunction name="initTestCase"> <Incident line="0" file="" type="pass"/> <Duration msecs="17.570825"/> </TestFunction> |
| testrow | Testrow for its parent testfunction (like 'WithoutProxy:ok01'), parsed from XML reports:
<TestFunction name="connectToHost"> <Incident line="0" file="" type="pass"> <DataTag> <![CDATA[WithoutProxy:ok01]]> </DataTag> </Incident>... |
TABLE. Database tables for 'fixed' data
| Table | Purpose |
|---|---|
| project_run | Project build data. On most Qt Metrics 2 pages only the Qt5 state build data is shown (unless stated otherwise on the page info) |
| conf_run | Configuration build data for its parent project_run |
| phase_run | Configuration build phase data for its parent conf_run |
| testset_run | Testset result data for its parent conf_run |
| testfunction_run | Testfunction result data for its parent testset_run, parsed from XML reports:
<TestFunction name="initTestCase"> <Incident line="0" file="" type="pass"/> <Duration msecs="17.570825"/> </TestFunction> |
| testrow_run | Testrow result data for its parent testfunction_run, parsed from XML reports:
<TestFunction name="connectToHost"> <Incident line="0" file="" type="pass"> <DataTag> <![CDATA[WithoutProxy:ok01]]> </DataTag> </Incident>... |
TABLE. Database tables for build data
| Table | Purpose |
|---|---|
| db_status | Information on database updates by the parser |
TABLE. Database tables for other data
DEVELOPMENT AND MAINTENANCE
This chapter describes a few typical development and maintenance cases as a supplement to the detailed description in the previous chapter. The SW designers who are maintaining and developing the system are the target audience.
Init Parameters
Certain behavior can be adjusted with the parameters in the ini file. An example version with the defaults is available with the source code (see below), but an actual version must be stored and edited manually into the Qt Metrics root directory.
The following can be adjusted:
- Number of last days to be included into top failures / flaky testsets / blacklisted list
- Number of testsets to be included into top failures / flaky testsets list
- Number of last days to be included into top duration list
- Duration limit (seconds) for top duration list (so that only the higher are listed)
Changing a 'number of days/testsets' value will have an impact how quickly or slowly any possible changes or improvements are visible in the lists. Please also note that increasing any value will have some impact to the performance how quickly the related pages and data will be opened.
Source Code and Directory Structure
The Qt Metrics code is available in code.qt.io (or GitHub). The list of current directories are shown below.
| Directory/File | Content |
|---|---|
| (main folder) | Common implementation files (e.g. index.php and the ini file) |
| fonts/ | Local custom fonts |
| images/ | Local images like logos and icons |
| lib/ | Components used by the Report Builder |
| scripts/ | JavaScript (and other) scripts |
| src/ | Model and controller classes |
| src/test/ | Unit test classes and SQL scripts used in unit and UI testing |
| styles/ | Style sheets |
| templates/ | View files (HTML files) |
TABLE. Directories
Security and Data Validation
Source data and validation
The source data used in the Qt metrics system is publicly available. The log files for the CI metrics page, for example, can be found in http://testresults.qt.io/ci/. Hence the data itself does not need any user access control.
The parser takes care of data validation before storing data to the database.
Database user access
The report builder of the Qt metrics system uses one user id to read the data from the database. This user has only a read access to the database. The parser uses another user id with write access to store the data to the database. A separate admin user is used for the administration actions, which are authenticated with HTTP basic authentication (see separate chapter).
The database management system cannot be accessed outside the testresults server, where both the report builder and the parser reside.
Web security
The following actions were taken to secure the web site.
The SQL injection is tackled by (as guided in php.net)
- Using customized user with limited (read) privileges (implemented in qtmetrics.ini)
- Using prepared statements with bound variables provided by PDO (implemented in Database.php)
- Using .htaccess to redirect any url to the index.php by defining a RewriteRule
- Using REST API via index.php which lists all the available methods
- Not keeping the authentication credentials in the database (ini file instead)
The Cross site scripting (XSS) is tackled by (as guided in phpsec.org)
- Using the strip_tags PHP function to clear possible scripts injected into input values (in index.php)
To ensure that the uninitialized variables cannot be accessed from the URL the register_globals directive is disabled (disabled by default in PHP versions 4.2.0 and greater)
Applied CSP and other security headers (in .htaccess)
- Apply Content Security Policy, affecting for example that
- all JavaScript code needs to be in separate files (instead of embedded inside the <script> tags in HTML files)
- the external components need to be included locally (in lib folder, instead of using them from CDN)
- Provide protection against MIME type confusion attacks
- Provide protection against clickjacking
- Provide protection against Cross-Site Scripting
HTTP Basic Authentication for selected routes (in HttpBasicAuthRoute.php)
Performance
Database Indexing
Each SQL query used needs to be tested for its performance (preferably in the target server with actual data and data size) e.g. with the EXPLAIN statement. The indexes must be created online to the database for existing tables. In addition, the queries to create the indexes must be added to the parser because (possibly) missing tables are created by it, as well as into the SQL script (*) used in unit and UI testing.
(*) See the src/test/qtmetrics_create.sql for the current indexes (or from the database directly if you can access it).
HTML Compression and Minification
The HTML compression is enabled in .htaccess for selected media types.
The template files do not use leading spaces to indent the HTML script for its hierarchy (the Twig script is indented normally as it does not appear in the generated code sent from server to the client). Minified versions of the included CSS files are used for the library components.
Site Map
The site map graph has been made with www.draw.io. The source xml file (site_map.xml) is located in the images directory. To save the image, use export as picture (png). To save the source xml, use export as xml (normal).
Maintaining the Version Info
The report builder version is visible via the About header menu item (about.html) and it shall be updated on every change affecting the UI.
Requests for New Data
New fields should be added to the database only with proper reason and use case. Only the parser adds data into the database. Therefore, if new data fields are needed to the database while specifying or implementing new content of the existing or new metrics boxes or pages, a change request to the parser is needed as well. The change request should include information as to whether the new data is already available, for example in some log files, or whether the change requires a change to the log files.
Development Environment
The development of the Qt metrics system can be done using any available web development environment that supports PHP and MySQL/MariaDB, like XAMPP, LAMP, WAMP or Cloud9 cloud IDE for example. The Qt Metrics code is available in code.qt.io (or GitHub). Cloning the Qt Metrics code does not include the external library components.
Library components to fetch from GitHub
Most of the external library components are available as Git submodules. In order to include them into the Qt Metrics code, please enter the following in the root sysadmin directory:
git submodule init
git submodule update
The first will write the required definitions into sysadmin/.git/config file, and the latter will clone the submodules to the lib directory.
Library components to fetch manually
The following external library components need to be fetched from their source location:
| Component | Location | Version | Destination (create folder) |
|---|---|---|---|
| jQuery | http://jquery.com/download/ | jquery-1.11.3.min.js | copy file to lib/jQuery |
| jQuery UI | http://jqueryui.com/download/all/ | 1.11.4 | unzip all to lib/jQuery-UI |
| jQuery UI themes | http://jqueryui.com/download/all/ | 1.11.4 | unzip all to lib/jQuery-UI-themes |
Init parameters
Please copy the qtmetrics_example.ini as qtmetrics.ini and edit the following into it (change as needed by your environment):
- Database connection data exactly like it is to be given to PDO.
dsn = "mysql:host=0.0.0.0;port=3306;dbname=qtmetrics;charset=utf8"
username = "root"
password = ""
username_admin = "root"
password_admin = ""
The database user for "username" must have read privileges, and the user for "username_admin" both read and update privileges.
In addition, in order to use and test the admin functionality edit the following (change as you like):
; admin credentials
admin_username = "name"
admin_password = "pw"
Database
The database and needed tables and indexes can be created with the SQL script src/test/qtmetrics_create.sql.
Sample data can be stored with the SQL script src/test/qtmetrics_insert.sql.
The tables and the data can be removed with the SQL script src/test/qtmetrics_drop.sql.
The data can be reloaded into the database by running the drop, create and insert scripts in a row.
Unit testing
Install the PHPUnit for unit testing the classes. The test classes are in src/test directory. These tests use the sample data in the database, so please reload fresh data (as described above) before running the tests.
The unit tests can be run in the root Qt Metrics folder (where the index.php is located), for example:
php <path-to-phpunit>/phpunit.phar ./src/test
php <path-to-phpunit>/phpunit.phar ./src/test/DatabaseTest
The first will run all test classes while the latter will run only a single test class.
Ajax functionality
In usability point of view it is a good practice that if loading a page will take more than a few seconds the user needs to be provided some feedback to the operation he/she requested. Retrieving data for some Qt Metrics pages (e.g. top lists, duration lists, blacklisted lists) may take some time (up to a minute or so). Therefore the pages have been built to use Ajax to provide indication that loading the data is in progress.
The 'normal' pages use typically an approach where one route reads the data and calls one view file in one go. The Ajax functionality is implemented by splitting this into two; first the 'main' route will show all the page elements in one view file (a progress bar in the element where data will be inserted into), and then the Ajax implementation will send a request to the 'data' route to retrieve the data (HTML formatted table). These two actions happen at the same time. The Ajax implementation will replace the progress bar with the actual content when the ready response is received later on. The Ajax implementation is in the scripts/ajax.js and the routing naturally in the index.php.
Submitting Code
The implemented open source code is made available to the master branch of the qtqa/sysadmin Qt repository in the sysadmin/non-puppet/qtmetrics2 folder. The submissions follow the common Qt contribution guidelines, including the use of the Gerrit Code Review tool.
Quality Assurance
The following actions should be taken to ensure the quality of the change commits.
Qt coding conventions
Use the Qt Commit Policy, Qt Coding Style, and Qt Coding Conventions where applicable for the PHP, JavaScript, HTML and CSS implementation, plus others if used.
Commits
As a general rule for all Qt contributions, it is important to make atomic commits. That means that each commit should contain exactly one self-contained change. Unrelated changes, or new functionality and error corrections, must not be mixed. Likewise, unrelated small fixes, for example to coding style, should not be hidden in bigger commits, but should be handled in a separate commit.
The Gerrit Early Warning System will make certain checks to the code. It is a good practice to make the check before pushing the changes to Gerrit to avoid unnecessary fix patches. The sanitizer tool, a script made with Perl, can be used for that.
Validating the scripts
It is recommended to follow the HTML5 standard and use the HTML5/CSS3 approach, for example to make the style definitions in the CSS file instead of in the PHP or HTML code. Please utilize the HTML markup validator and the CSS validator by the World Wide Web Consortium (W3C). In addition, use for example JavaScript Lint to validate JavaScript code.
Unit testing
The development of the Qt metrics system can be done using any available web development environment that supports PHP and MySQL/MariaDB, like XAMPP, LAMP, WAMP or Cloud9 cloud IDE for example. PHPUnit shall be used for unit testing the classes (test classes are in src/test directory). For that some sample data shall be stored into MySQL/MariaDB database tables (SQL scripts in src/test directory). New functionality or content may require modifications or additions to the sample data.
Verification on target server
The changes can be verified on the testresults server in a specific verification folder (http://testresults.qt.io/qtmetrics-dev/new) before submitting changed code. The CI team will assist with the access rights.
Verification with different browsers
Verification in all major browsers is essential for any web based service. The Qt metrics system follows responsive design with Twitter Bootstrap to adjust to different screen resolutions. However, it must be noted that, although the pages may utilize large screens, the visible content area should be defined to be as narrow as possible and the most important content to be placed on the left to prevent unnecessary horizontal scrolling.
All changes should be verified at least with the following browsers: Chrome, Firefox, Opera, IE and Safari. Use of different operating systems and browser versions, as well as mobile browsers, is also recommended. It is proposed to list the verified browsers in the Gerrit commit description.
Qt brand guidelines
The Qt Brand Guidelines shall be followed where applicable.
First Aid
Below proposed solutions for some maintenance related issues. These actions can be done CI team or someone with the required privileges.
| Problem | Solution |
|---|---|
| The number of testsets in the top and other lists is either too low or too high | Change the definitions in the ini file |
| Opening the pages becomes slower and slower | Remove old data (the oldest dates) by using the admin interface |
| The build/test result tables have become too large (they include too old data) | Remove old data by using the admin interface |
| There is an old branch visible which is no longer needed | By using the admin interface, either archive the branch (data will remain in the database and can be restored) or remove it (data will be deleted from the database) |
| New branch does not appear in the Results in Branches section on the Qt5 page (http://testresults.qt.io/qtmetrics-dev/new/buildproject) | Is there a new configuration and has it been added to the configuration list of the parser (old-CI) |
| New configuration does not appear on the build lists | Has it been added to the configuration list of the parser (old-CI) |
| Test results do not appear on testset pages (for a new branch or configuration) | Are the test related tags the same as for other branches/configurations (e.g. the begin tag “QtQA::App::TestRunner: begin tst_some_testset”) |
FUTURE DEVELOPMENT
The target is that anyone from the Qt developer community could contribute to the development and improvement of the Qt Metrics page.
Backlog
The development items are listed in the:
- Qt Metrics 2 backlog: Qt Metrics 2 Backlog
SUPPORT
In case of questions, proposals, and improvement ideas, please contact the development team:
- Juha Sippola (Developer: report builder, database)
- Tony Sarajärvi (Developer: parser; Use case designer, Key user, Reviewer)
- Frederik Gladhorn (Use case designer, Key user, Reviewer)
- Simon Hausman (Key user, Reviewer)
- Oswald Buddenhagen (Key user, Reviewer)
- Friedemann Kleint (Key user, Reviewer)
You may also leave comments into Qt Forum.