View source for Sanity-Test-Guidelines


[[Category:Release::Testing]]


= Sanity Test =

Sanity Test is a part of the releasing cycle. The purpose is to ensure that the Qt SDK is ready for further testing.

<span style="color:blue">'''RTA work ongoing:''' </span>
Part of the Sanity test cases are covered by Release test automation (RTA). See more information on the <span style="color:blue">'''RTA'''</span> (covered by RTA) and <span style="color:red">'''RTA'''</span> (not yet covered by RTA) sections below.

== Build checks ==

* '''Asks about license'''

<span style="color:blue">'''RTA:'''</span> RTA verifies that license key is asked while running the configure (valid license key, invalid license key). When the installer is run the installing with a license key is verified by RTA.

* '''Compiles with minimal options (eg. -opensource -confirm-license)'''
Run configure command with the following options
<code>
configure -developer-build -opensource -nomake examples -nomake tests -confirm-license
</code>
<span style="color:blue">'''RTA'''</span> Linux, Mac, Windows: covered by RTA

* '''Compiles as static build (where supported)'''
Run configure command with '-static':
<code>
configure -static …
</code>
<span style="color:blue">'''RTA'''</span> Linux, Mac, Windows: covered by RTA

* '''Compiles in namespace (where supported)'''
Run configure command with '-qtnamespace <namespace>':
<code>
configure -qtnamespace CustomName …
</code>
<span style="color:blue">'''RTA'''</span> Linux, Mac, Windows: covered by RTA

* '''Compiles with shadow build'''
Create a new folder. cd to the new folder. Run the configure command from the source directory.
<span style="color:blue">'''RTA'''</span> Linux, Mac, Windows: covered by RTA

* '''Compiles cross-compiled'''
Use -xplatform parameter, e.g.
<code>
configure -embedded arm -xplatform qws/linux-arm-g++ …
</code>

== General checks ==

* '''Text files have the correct EOL'''
Open a few text files (README, .cpp, .h, .pro, etc.) and check if they look OK.
* '''Files/dirs have sane perm. and timestamps'''
* '''Tags have been replaced properly'''
The list of tags: qmake -query (???)
* '''README has valid information'''
Complete information on how to build Qt from sources should be given.

== Installer checks ==

* '''Right version of Qt Installer Framework is used'''
Launch the installer with '—version' . For Qt 5.4 -beta it should show 'IFW Version: "1.5.82", Installer base SHA1: "f67db7e",
<span style="color:blue">'''RTA:'''</span> Linux, Mac covered by RTA.
<span style="color:red">'''RTA:'''</span> Windows: Manual testing required.

* '''Fresh install works correctly'''
Install Qt SDK on a newly installed OS.

<span style="color:blue">'''RTA:'''</span> RTA verifies that clean installation works ok. Clean means that the installation directory does not contain any other Qt version.
<span style="color:red">'''RTA:'''</span> Whole OS is not installed as a fresh installation during the RTA. Study how to do this added into CI's ta

* '''Fresh install to place requiring elevated rights works correctly'''
Install Qt SDK as a normal user to a place requiring elevated rights. The installer should allow this by asking for elevated rights / the sudo password.

* '''Upgrade install works correctly'''
Install higher version of Qt having older one, e.g. install Qt 5.3.2 on Qt 5.2.0.

* '''Parallel install works correctly'''
Install two Qt installation in parallel, e.g. Qt 4.8.6 and 5.3.2. Check that both packages work
* '''Warns when installing over an installation'''
Run the installer the second time. A warning pop-up windows should be displayed.

* '''Is correctly signed (vendor/certificate, not untrusted)'''
Check where applicable.

* '''Displays appropriate gfx & strings & version no.'''
This check is about general look and correct package version number.

* '''Offers the correct license(s)'''

<span style="color:blue">'''RTA:'''</span> RTA verifies that installation works with the valid license key.
<span style="color:blue">'''RTA:'''</span> Visual verification of the installer and installing with the invalid license keys needs to be done manually.

* '''Offers sane default install directory (w/version number)'''
Check of the default folder is similar to ../Qt5.3.2/
<span style="color:blue">'''RTA:'''</span> RTA verifies the default directory.
* '''Correctly installs to default directory'''
<span style="color:blue">'''RTA:'''</span> RTA verifies installing into the default directory.
* '''Correctly installs to non-default loc.'''
Change the target folder during installation from ../Qt5.3.2/ to a different name.
<span style="color:blue">'''RTA:'''</span> RTA verifies installing into the non-default directory.
* '''Sanely reports progress and completion'''
Click on the Details button to see the progress. Wait for successful completion of the installation process.

* '''Installs only selected components'''
Try to use Deselect All button, and select a few additional components. Component selector should works sanely. After installation is done, only selected components should be in the target folder.
<span style="color:blue">'''RTA:'''</span> RTA verifies that only selected components are installed.
* '''Shortcuts from last page of installer works'''
Select/deselect two check boxes (to run Qt Creator and see notes) on the final screen.
* '''Correctly creates Desktop shortcuts'''
* '''Correctly creates Start Menu shortcuts'''
Check if Qt Creator can be found by the OS (start menu, launcher, dash home, etc.). Check the icon.
* '''Sets correct environment variables for shell'''
* '''Package shows up in Control Panel/Pkg. Manager'''
* '''Patching of files is done correctly'''

* '''Cancel button is available'''
Use cancel button. Check if there are no changes done by the installer when the installation is interrupted.
<span style="color:blue">'''RTA:'''</span> RTA verifies that the installation can be cancelled. Note: RTA does not support cancelling the installing after the 'install' button is pressed i.e. while the installing of the components is ongoing.
* '''Cleans up after install failure'''

* '''Cleanly fails on insufficient disk space'''

== Uninstaller checks ==

* '''Uninstalls properly'''
<span style="color:blue">'''RTA:'''</span> RTA verifies that uninstalling is done properly.
* '''Removes installed files properly'''
<span style="color:blue">'''RTA:'''</span> RTA verifies that all installed files are removed.
* '''Removes empty directories'''
<span style="color:blue">'''RTA:'''</span> RTA verifies that the installation directory is removed.
* '''Removes registry keys'''
* '''Reverses any other changes'''

== Both source/binary package checks ==

* '''License is correct'''
<span style="color:blue">'''RTA:'''</span> RTA verifies that proper license files are found from $QTDIR\Licenses directory.
* '''Assistant works correctly'''
Run Assistant (e.g. ./5.0.2/gcc_64/bin/assistant). Check the basic functionality, browse through, search for documentation.
<span style="color:blue">'''RTA:'''</span> RTA verifies that Assistant can be opened & closed, browsing and searching works.
* '''Designer works correctly'''
* '''Qt Creator works correctly'''
Qt Creator can be tested using the following test cases: [http://wiki.qt.io/Category:Tools::QtCreator::QualityAssurance Qt Creator Tests:]
* '''Demos and examples launch without crashing'''
<span style="color:blue">'''Each example'''</span> should be run with no error.
<span style="color:blue">'''RTA:'''</span> RTA launches and closes each Qt example (except console examples).
* '''Demos and examples function acceptably'''

'''RTA:''' Sanity level testing; see the QtModule sections below.

* '''Demos and examples can be rebuilt'''
<span style="color:blue">'''RTA:'''</span> RTA compiles the examples and demos from /Src and /bin directories.
* '''External apps build (Qt Creator and other applications)'''
Create your own application, compile it and run.

* '''"DLL Swapping" work on an application'''

* '''GUI stress-testing works ok'''
Try to perform many keystrokes, mouse clicks, etc.
* '''Audio/Video w/QtMultimedia works'''
* '''Raster engine works'''
Run ./qtbase/examples/gui/rasterwindow example.
* '''Image formats work'''

* '''GraphicsView works'''
* '''OpenGL works'''
Run ./qtbase/examples/gui/openglwindow example.
This is also checked by most of the other examples.
* '''Printing works'''
* '''QML 1 apps work (qmlviewer)'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtQML1 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QML 2 apps work (qmlscene)'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtQML2 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtNetwork works'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtNetwork examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtSql works'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtSql examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtSvg works'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtSvg examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtXml works'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtXML examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.

* '''QtWebEngine works'''

* '''QtWebKit 1 works'''

* '''QtWebkit 2 works'''

* '''Qt Script 2 works'''
Run ./qtscript/examples/script/qstetrix example. Use Debug option. context2d can be also run.
<span style="color:blue">'''RTA:'''</span> RTA runs few QtScript examples and verifies that some detailed level Qt functionality can be used. Example ./qtscript/examples/script/qstetrix is included into the testing. They are launched via the command prompt i.e. not from the Qt Creator.

== Qt Creator Checks ==

* '''Right Qt Creator version is installed'''
Check Qt Creator version in About dialog.

<span style="color:blue">'''RTA:'''</span> Linux, Mac: Qt Creator version is verified from the About dialog.
<span style="color:red">'''RTA:'''</span> Windows: not yet part of RTA (some testing tool issues; work ongoing to solve this). Manual testing required.

* [http://wiki.qt.io/Category:Tools::QtCreator::QualityAssurance Qt Creator Tests:]

<span style="color:blue">'''RTA:'''</span> Linux, Mac: Qt Creator is opened and closed in RTA tests.
<span style="color:red">'''RTA:'''</span> Windows: Qt Creator open & close not yet covered by RTA (some testing tool issues; work ongoing to solve this). In general, other test cases except open&close are not yet part of RTA. Manual testing required.

== CMake Checks ==

Ensure that the binary package was installed with the ‘source components’ checked.

On Linux, install cmake from a repository (at least CMake version 2.8.3) and copy a script like this to your Qt installation and run it:

<code>#!/bin/bash

QT_INSTALLATION_PREFIX=$PWD/gcc_64

for d in `find . -name "cmake.pro"`; do
 b=`dirname $d`

m=`dirname $b`
 m=`dirname $m`
 m=`dirname $m`
 m=`basename $m`
 if [ x$m != x"qtactiveqt" ]; then
 pushd $b > /dev/null
 rm -rf build
 $QT_INSTALLATION_PREFIX/bin/qmake
 make check
 popd > /dev/null
 fi
done

</code>

On Windows with msvc use a bat script instead (download cmake from http://www.cmake.org/cmake/resources/software.html ):

<code>

set QT_INSTALLATION_PREFIX=%cd:%/msvc2010
set scriptbase=%cd%
FOR %%? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (
 cd Srcamp;#37;%?rd /S /Q build
 %QT_INSTALLATION_PREFIX%/bin/qmake
 nmake check
 cd %scriptbase%
)
cd Src\qtwebkit\Source\WebKit\qt\tests\cmake
rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
nmake check
cd %scriptbase%

</code>

If testing with MinGW on Windows, use a script like this;

<code>

set QT_INSTALLATION_PREFIX=%cd:%/mingw48_32
set scriptbase=%cd%
FOR %%? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (
 cd Srcamp;#37;%?rd /S /Q build
 %QT_INSTALLATION_PREFIX%/bin/qmake
 mingw32-make check
 cd %scriptbase%
)
cd Src\qtwebkit\Source\WebKit\qt\tests\cmake
rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
mingw32-make check

</code>

On all platforms, the result should be output showing the cmake tests running and passing. Please report failures to the releasing mailing list.

= Links =

* [http://wiki.qt.io/Building-Qt-5-from-Git Building Qt 5 from Git]
* [http://www.qtcentre.org/wiki/index.php?title=Building_static_Qt_on_Windows Building static Qt on Windows]
* [http://qt.io/faq/answer/what_is_the_usage_of_the_qtlibinfix_and_qtnamespace_configuration_options What is the usage of the qtlibinfix and qtnamespace configuration options?]
* [https://bugreports.qt.io/secure/Dashboard.jspa Bugtracker]