Network Testing

We can now test networking code using local containers to run the test servers one needs. This shall make it possible for all contributors to test changes to network code. For the moment, all existing network test code runs against a test server that's not easy to set up.

Overview docker-based test servers

The container-based solution shall launch itself transparently when doing make check, provided the relevant infrastructure is available. If the user has installed docker engine on the host, the test servers will be built up inside separate docker containers, and then, the test case goes with the docker-based test servers. Otherwise, the test case will keep using the legacy QTest server (qt-test-server.qt-test-net).

Error creating thumbnail: File missing

System Overview

Error creating thumbnail: File missing

Sequence Diagram

Target systems

1. Linux (Ubuntu)
2. macOS (Darwin)
3. Windows 10 (x86_64)

System Requirements

1. docker-compose version > 1.18.0 - Added support for --timeout in docker-compose down
2. mDNS – Windows Bonjour Printer Services; Linux avahi-daemon, avahi-dnsconfd, resolvconf

Implemented features

1. Integrate docker-compose into make check transparently for network-related Qt autotests
2. Split the legacy QTest server into multiple docker containers
3. Create new containers and tidy away related files for each test
4. Using mDNS to discover servers in the bridge network
5. To support Qt Coin framework, the files used for provisioning are saved in qt5
6. Provide an internal testserver.pri for Qt autotests. Therefore, the tests can be replaced one by one.

NOTE: Why do we adopt docker-machine for both macOS and Windows

There are some known limitations of "Docker for Mac" and "Docker for Windows". The tests running on the host can't directly connect to the containers.

For example, (as mentioned in "Known limitations, use cases, and workarounds" of Docker docs)

1. There is no docker0 bridge on macOS / Windows.
2. I cannot ping my containers
3. Per-container IP addressing is not possible.

Docker docs recommends using port mapping as a workaround. The side-effect is that it causes a port conflict if the user is running a service that binds the same port on the host. An alternative solution is to deploy the docker environment into VirtualBox using docker-machine.

https://docs.docker.com/docker-for-mac/networking/

https://docs.docker.com/docker-for-windows/networking/

NOTE: In addition, Docker for Windows (Hyper-V) doesn't work on Windows 7 and Windows 10 home edition

Prerequisite

The user must run the provisioning scripts in advance before attempting to build the test servers. The docker_testserver script is used to build up the docker images into the docker-cache. It handles the immutable parts of the server installation that rarely need adjustment, such as downloading packages.

Install Docker packages

Download and install the docker engine, docker-machine, and docker-compose. There are shared scripts used by Qt Coin. If you can`t find a suitable one for your system, please check the online docker docs.

1. qt5/coin/provisioning/common/linux/docker.sh
2. qt5/coin/provisioning/common/macos/docker.sh
3. qt5/coin/provisioning/common/windows/docker.ps1

NOTE: For Linux platforms, remember to start a new login session after Docker installation (su $USER).

Provisioning test servers

Before starting the test servers, it requires the user to run the provisioning script in advance. When building the images, the script uses the SHA-1 of each server context as the tag of docker images. A tag labels a specific image version. It is used by docker-compose to launch the corresponding docker containers.

1. Ubuntu: sudo qt5/coin/provisioning/common/shared/testserver/docker_testserver.sh
2. macOS: qt5/coin/provisioning/common/shared/testserver/docker_testserver.sh VMX
3. Windows: & 'C:\Program Files\Git\bin\bash.exe' --login qt5/coin/provisioning/common/shared/testserver/docker_testserver.sh VMX

You can run the command (docker images -aq "qt-test-server-*") to list all the tag of server images.

                                                                                         
REPOSITORY               TAG                                      IMAGE ID                             
qt-test-server-danted    2d5eea7356dd0ba25f3c12d1cba00d70770da1b0 9106f1fc3449
qt-test-server-ftp-proxy 2c6c8f1ab6a364b540c43d705fb6f15a585cb2af e8134fd91d2e
qt-test-server-vsftpd    18896604c7e90b543e56d80c8a8aabdb65a590d0 8c1a4bd4623c
qt-test-server-squid     577d99307eea9a8cccfec944d25be2bce2fe99cc ca4e09991cc8
qt-test-server-apache2   cc9ea678b92bdda33acb9fa0159bb4ad0f3cd947 620c4ed46911

How to convert a test to use the provisioned test servers

Here is an example of reworking QNetworkReply tests.

Some tests still need to support the legacy QTest server (qt-test-server.qt-test-net) before the server is sunset. There are some helper functions help you to get the hostname of the provisioned servers. If you can't find a suitable one in this file, please create it and set the corresponding hostname defined in the docker-compose file.

qtbase/tests/auto/network-settings.h

static QString httpServerName()
{
#ifdef QT_TEST_SERVER_NAME
    return QString("apache2.") % serverDomainName();
#else
    return serverName();
#endif
}

Add a list of server names to QT_TEST_SERVER_LIST

Each test can define their depending servers. The server name shall be defined in the docker-compose file. The docker engine then brings up the servers before running the test, and shuts them down afterwards.

qtbase/tests/auto/network/access/qnetworkreply/test/test.pro

QT_TEST_SERVER_LIST = vsftpd apache2 ftp-proxy danted squid

Include testserver.pri in the project file

The testserver.pri file adds support for this feature built by Qt Autotest projects that need them. This instructs qmake to insert additional targets into the generated Makefile.

qtbase/tests/auto/network/access/qnetworkreply/test/test.pro

                                           
include($$dirname(_QMAKE_CONF_)/tests/auto/testserver.pri)

Rework test case with the predefined macro

QT_TEST_SERVER is a predefined macro used to check whether docker-based servers are in use and change the test parameters at compile time. Avoid using QtNetworkSettings::serverName(), and then use the helper functions declared in network-settings.h instead. To make sure that all servers are ready, use QtNetworkSettings::verifyConnection() with the server names and their opening ports in initTestCase().

qtbase/tests/auto/network/access/qnetworkreply/tst_qnetworkreply.cpp

void tst_QNetworkReply::initTestCase()
{
#if defined(QT_TEST_SERVER)
    QVERIFY(QtNetworkSettings::verifyConnection(QtNetworkSettings::httpServerName(), 80));
#else
    if (!QtNetworkSettings::verifyTestNetworkSettings())
        QSKIP("No network test server available");
#endif
}

How to create a new test server

You need to create (or extend) a Dockerfile for the new server, and then add a service in docker-compose file. It tells the docker engine how to build up the docker image and the container. In qtbase, a shared docker-compose file is located in the tests folder.

qt5/coin/provisioning/common/shared/testserver/"NEW-SERVER"/Dockerfile
https://docs.docker.com/engine/reference/builder/

qtbase/tests/testserver/docker-compose.yml (or docker-compose-for-macOS.yml / docker-compose-for-windows.yml)
https://docs.docker.com/compose/compose-file/

Append the server name "NEW-SERVER" to the list of settings.

qt5/coin/provisioning/common/shared/testserver/settings.sh

Create an initial script and put the related test data to qtbase/tests/testserver/. When docker-compose launches the container, this script is used to configure and start the service. The test data will be mounted to $TESTDATA (read-only) at run time.

qtbase/tests/testserver/"NEW-SERVER"/"NEW-SERVER".sh
qtbase/tests/testserver/"NEW-SERVER"/testdata/