CMake Port/Development Notes

2019

01. August 2019

- Lots of work happening regarding properly building qml modules
- Initial work done on porting qtquickcontrols2
- Initial work done on getting the Qt Quick Compiler (qmlcachegen) to be used when building qtdeclarative / qtquickcontrols2 
- Implemented proof of concept for building universal (device and simulator) libraries for iOS, confirmed working with qtbase, qtsvg, qtdeclarative, but the work requires upstream changes in CMake and vcpkg (which is not merged yet)
- Looooots of work trying to get the wip/qt6 branches of all repositories into shape and getting the qt5.git wip/qt6 branch be up-to-date
- Some small fixes done regarding static builds, static plugins and cross-compilation
- Fixed a small bug in the pro2cmake conversion script to handle APPLE conditions correctly

Other interesting news:

- Qt Creator master branch learned to use the new CMake File API, which allows nicer integration with Qt Creator's project explorer
- Upstream work being done by Qt developers in CMake to get official CMake precompiled header support (it's almost done) and Unity builds (WIP)
- CMake precompiled header support confirmed to reduce build times of Qt Creator when built with CMake

04. July 2019

- Most CI code to support CMake has been done and merged to master, but needs to be deployed to production Coin, and then we would need to start a multi step process to make CMake and qmake builds coexist
- Lots of work done to support Android builds, building Qt libraries and applications using Qt libraries has been tested to work on armv7 devices and emulator
- Some preliminary work done to provide a convenience toolchain for building other modules and applications, which is especially relevant for Android / embedded Linux
- Lots of fixes for embedded Linux cross-builds regarding Qt Tools and packages
- Added ICC specific fixes to build on CI
- Added support for vcpkg toolchain chain loading in the Qt generated toolchain file

The rough steps that were considered for preparing CMake and qmake co-building in CI are:

- Update qt5 repo wip/qt6 branch to contain UseLegacyInstructions flag for all qmake configs, this is just a preparation step and everything should be built as before
- Merge all wip/cmake branches into wip/qt6 branches of all submodules (qmake configs still build as before, no new cmake configs are built yet, but this does bring in the yaml files from the wip/cmake branch) 
- We need to update qt5 repo wip/qt6 branch to contain all updated SHA1s of wip/qt6 repos (this will probably be the biggest issue, possible solutions is to wait for the pinning bot, or use direct pushes)
- Activate the cmake based configurations in qt5 wip/qt6 by duplicating the qmake configurations, and just removing the UseLegacyInstructions flag, but only for specific platform configurations that are known to be built in the wip/cmake branch 

13. June 2019

- 2 day workshop happend on 4th and 5th of June
- More work done on CI side to make Windows and macOS early warning system work
- qtbase dev was merged into wip/qt6, in preparation of merging wip/qt6 into wip/cmake
- More work done on porting qtdeclarative, libraries and tools are built, started working on qml plugins
- Ported WinMain for Windows gui applications
- Work started on Android support, qtbase already builds, the android support library is built, and necessary bits for deployment (generation of json file) are also available
- Added basic support for compiling applications using a qmake built with CMake
- Added warnings_are_errors support
- Fixed test data to be found properly, so now more tests should pass
- Implemented standalone tests building (needed for CI)
- Implemented standalone examples building (needed for end-users and Qt Creator)
- Ported QtNetworkAuth (so now we have qtbase, parts of qtdeclarative, qtsvg, qtimageformats, qtnetworkauth)
- Fixed some Windows issues due to bit rot
- Implemented initial support for automatic registration of Qt plugins to modules, and started work on automatic inclusion of plugins in static builds (patch in review)
- Fixed some xcb issues
- Implemented proper exception support
- Fixed OpenGL related issues
- Started work on implementing super builds (qt5 build), patch in review
- Fixed WinMain to be automatically linked into GUI applications

23. May 2019

- Coin now tests every single patch pushed to Gerrit to the qtbase wip/cmake branch on a single Linux Configuration, and posts the results on the Gerrit change (not enforcing), work is being done to test on Windows and macOS as well
- Most of qtimageformats has been ported
- Tests are now buildable with an installed Qt (useful for Coin)
- Implemented bundled double-conversion building (so using a package manager to install it is not strictly necessary anymore)
- Fixed hard requirement on needing Widgets when building any other module (was needed for qtimageformats)
- Fixed qmake parser to handle qmake syntax better in pro2cmake.py (doesn't crash anymore when trying to run it on qtdeclarative files )
- Continued investigation into building documentation and translation files for Qt Creator with CMake, once done the work would be forwarded to the Qt CMake Port 
- Work is being done on test data installation, so that tests that use test data can now pass

16. May 2019

- Implemented non-prefix builds (enabled with developer-build)
- Ported a part of qtimageformats
- Side work finished on porting Qt Creator to build with cmake, building docs for Qt Creator might be used for qtbase and friends
- Configure script changes merged to allow passing arbitrary cmake flags
- Continued looking into .pri file generation
- Various small fixes

09. May 2019

- Implemented script to keep manual modifications when regenerating projects
- Fixed some issues regarding DBus 
- Started looking into non-prefix builds (developer-build)
- Continued work from workshop on library mapping,
- Modified conversion script to support standalone example building
- Fixed various Linux package issues
- Regenerated a bunch of subprojects in src/
- More work on configure argument mapping
- Started looking into .pri file generation for built modules (so that qmake can use them while building apps or other qt modules)
- Continued working on the version-less targets
- Continued work on static plugins auto importing

03. May 2019 Workshop day 2

Things implemented during workshop

1. Final touches on packages providing tools
2. Final touches on keeping track which dependencies need to be found while consuming Qt in a CMake application
3. Fix current branch to be buildable with released CMake 3.14.x
4. Added a few missing xcb platform plugins
5. Implemented building qtsvg against qtbase
6. Support building qtbase & qtsvg as static libs
7. Fixed building offscreen platform plugin
8. Improved GLX detection and GL integration
9. Improved convertor script and regenerated configure.cmake files to keep track of more 3rd party dependencies
10. Updated cmake/README
11. Fixed finding host tools during cross compiling
12. Implemented exporting of major-version-less Qt targets, to facilitate easier upgrading of CMake applications from Qt5 to Qt6
13. Improved mapping of libraries from qmake to CMake land
14. Improved handling of _nolink targets where only include headers have to be propagated (Vulkan, OpenSSL)
15. Investigated the possibility of providing a user friendly Qt + CMake toolchain file for CMake applications to consume
16. Initial fixes for building qtbase with MinGW
17. Started work on building examples as standalone projects (not part of qtbase build)
18. Tested a few successful qtbase builds on a private Coin instance
19. Started work on handling static plugins in a consuming CMake application (when using a static qtbase build)

02. May 2019 Workshop day 1

Discussion points

1. Summary on the current state of CI builds of Qt-CMake-port
2. What changes have been made since last workshop
3. Be able to build applications against Qt5 or Qt6 without doing a lot of search and replaces in project files
4. What branch we want to target with our CMake changes (wip/cmake or qt5/dev or qt6/dev)
5. Time line on when we think we could merge the CMake changes into the mainline branch (Milestone 1 probably)
6. Depending on which branch we merge, decide If we need to keep qmake build of Qt working – if merged to qtbase/dev, then we need to keep qmake anyway
7. The convenience of using CMake to build applications instead of qmake (simplify what the user has to specify on the CMake command line), if we can accomplish that easily
8. What package manager we want to use for 3rd party packages (vcpkg, Conan, both, something else?)
9. What we want to do with qmake (move into different repo, keep it in qtbase?), we need the mkspec files currently which are provided in qtbase/mkspecs.
10. Configure script for CMake (do we want it? What do we want it to do?)
11. How do we want to keep up to date with qtbase/dev (regeneration of project files)
12. Examples and tests (building outside of a Qt build)

CI CMake State of the Union

Qtbase can be built currently with dirty hacks on the 3 major configs (Windows, Linux, macOS)

Implementation to allow choosing between building using CMake and QMake Qt builds properly in Coin (using yaml files in project repositories). Allows specifying build steps and options.

Tests are currently built as part of the build VM.

Timeline is 1-2 months to get the proper Coin stuff in.

What changes have been made since last workshop

We decided to get rid of qrc files, we changed instead to generate CMake rules with the pro2cmake.py convertor, which reads the qrc files and then generates a file list of files that were inside the qrc file, which are then embedded in the CMakeLists.txt. We need to do it for all modules.

TODO: Backport the qrc file conversion to qt5.

Qt has public dependencies. In static builds of Qt, private dependencies become public. We need to propagate those inside the generated QtCoreConfig.cmake files, so that when an application does find_package(Qt5Core), it also tries to find Core's dependencies. We found a way to do that in a nicer way, intead of tracking the dependencies manually (as it is done currently in KDE).

Lots and lots of improvement in the pro2cmake and configurejson2cmake convertor scripts: qrc handling, path handling, simd sources, no_link targets (vulkan, openssl), public and private dependencies handling.

As a result we merged qt5/dev into wip/cmake, and regenerated the files. This got us much closer to the current state of qtbase/dev.

Sample of special casing:

add_qt_library(Foo  
    SOURCES  
        bla.cpp foo.cpp 
    LIBRARIES  
        ZLIB::ZLIB # special case 
        # FreeType special case : remove this 
) 

We now generate Config files for each module's tools, for example Qt5CoreTools exports rcc,moc tools,

Qt5WidgetsTools exports uic tool. These packages can be used to cross-compile Qt (for Yocto).

We also were able to do a Yocto cross build.

Configure script state

It now understands –D CMake arguments and forwards them to CMake, and also qmake --foo commands, but the –foo commands are currently not converted to CMake style –D arguments, so it's just for qmake Qt builds.

It currently does not support in-source builds, and instead creates build directories and invokes CMake there.

Windows configure script has not been changed yet.

The conigure script is not strictly necessary for the CI now (which was the initial reason why we wanted configure to map to CMake options) because of the YAML changes.

It's still useful to provide the convenience for the user (so existing configure lines don't berak), so the question now is when should we do it, and what mechanism (duplicate mappings in .sh and .bat files, unified CMake script, something else?)

Be able to build applications against Qt5 or Qt6 without doing a lot of search and replaces in project files

Allow mixing Qt5 and Qt6 packages in a project, to allow partial porting of projects.

# Usecase: application wants to mix both Qt5 and Qt6, to allow gradual porting 
set(QT_CREATE_VERSIONLESS_TARGETS OFF) 
find_package(Qt5 COMPONENTS Core Gui Widgets) # Creates only Qt5::Core 
find_package(Qt6 COMPONENTS Core Gui Widgets) # Creates only Qt6::Core 

# Usecase: application doesn't mix Qt5 and Qt6, but allows to fully switch to link against either Qt5 or Qt6 
set(MY_APP_USE_QT6 TRUE) # <- set at command line by application developer 
set(MY_APP_QT_MAJOR_VERSION 6) # <- set at command line by application developer
# set(QT_CREATE_VERSIONLESS_TARGETS ON) <- Default, doesn't need to be set 
if(MY_APP_USE_QT6) 
   find_package(Qt6 COMPONENTS Core Gui Widgets) # Creates Qt6::Core, Qt::Core 
else() 
   find_package(Qt5 COMPONENTS Core Gui Widgets) # Creates Qt5::Core, Qt::Core 
endif() 

Or you could use

find_package(Qt${MY_APP_QT_MAJOR_VERSION} COMPONENTS Core Gui Widgets) # Creates Qt6::Core, Qt::Core 

What branch do we want to use for the CMake port

Make a recommendation to merge to qtbase/dev on mailing list, and make it blocking, so either CMake or qmake changes will both block integrations if build or test fails.

Pros merging into qtbase/dev

• Supporting two build system will have to be done anyway, regardless of branch, so might as well to do it in dev: because it spreads knowledge (more exposure to CMake port), that we distribute the work of maintaining the CMake changes.
• Regardless of the branch it goes into, somebody would still have to maintain two build systems, either in dev or in wip/qt6, the question is just where the changes will have to be made. If it's done in wip/qt6 -> less people can do the work (because not many people are currently involved in that branch).

Cons merging into qtbase/dev

• If a Coin Cmake config fails, it will fail the qmake one as well
• More load on the CI
• People need to learn a new build system ( but let's be real, they'll have to do it at some point anyway )

Time line on when we think we could merge the CMake changes into the mainline branch: To be discussed, but probably not before the yaml Coin change goes into production, which would take 1-2-3 months, so it depends on Coin production update.

The convenience of using CMake to build applications instead of qmake (simplify what the user has to specify on the CMake command line), if we can accomplish that easily

Provide a toolchain file, that would ease use for CMake projects, make it as easy as using qmake for a project.

This includes passing along things from configuration time like: mkspec, address sanitizer flags, original toolchain file which was used while building Qt, the public dependency Find modules (ZLIB, Atomic), build type (debug vs release) on windows, because of different MSVC runtimes.

What package manager we want to use for 3rd party packages (vcpkg, Conan, both, something else?)

Qt wants a package manager for 2 reasons:

Building 3rd party packages (to get rid of src/3rdparty)

To replace IFW with something better / nicer

The consensus is that for replacing IFW, Conan is the better choice.

For acquiring 3rd party packages, vcpkg is a much nicer solution, because Conan has some issues with package name and target name generations

(Conan generates all lower case names, while vcpkg and upstream CMake uses upper case or mixed case).

Upstream Conan issues:

https://github.com/conan-io/conan/issues/4430

https://github.com/conan-io/conan/issues/4460

https://github.com/conan-io/conan/issues/4597

https://github.com/conan-io/conan/issues/3400

What we want to do with qmake (move into different repo, keep it in qtbase?), we need the mkspec files currently which are provided in qtbase/mkspecs

Three action points:

We need to clean up the mkspecs to be integrated with CMake (generate the platform headers) (timeline is "has to happen before 6.0")

We need to make qmake compile against Qt without ifdefs (timeline is "has to happen before 6.0")

Consider moving qmake out of qtbase (timeline is later than release of qt 6.0)

Pros for keeping qmake in qtbase

Coin would test if qmake can link against QtBase within a qtbase test run, which allows us to find issues with qmake ASAP as long as we support it.

Cons

Psychological argument that qmake is still around

Coin adjustments will be needed, if qmake is in a different repo

How do we want to keep up to date with qtbase/dev (regeneration of project files)

Depend on Coin to tell us if something breaks upon a dev -> wip/cmake merge.

Proposal to make merges every two weeks, to keep up to date with new changes.

When doing a merge, do a git diff, to see if any .pro and .pri and configure.json files were touched, which is a good indicator that some CMakeLists.txt files might need to be regenerated.

Examples and tests (building outside of a Qt build)

Tests.

First workable solution would be to build tests together with Qt in Coin, and then just copy all the generated and binary files under qtbase/tests to a different machine, keeping the exact same file paths.

Examples.

Rewrite pro2cmake.py generation script to generate cmake code for examples which would be standalone and not using QtBuild.cmake functionality. This implies not using add_qt_executable, and using regular add_executable() and macros that are exported (for dbus handling, rcc handling, etc).

25. April 2019

- First merge in a while was done from qt5/dev into wip/cmake and as a result regenerated important CMakeLists.txt files.
- Fixed Windows build after qt5/dev -> wip/cmake merge and file regeneration
- Improved the exporting Tools CMake Targets patch
- Implemented 3rd party library dependency propagation when exporting module config files (packages like ZLIB or double-conversion)
- Implemented dependency propagation for a static build of Qt (targets like tinycbor, qtharfbuzz, simd targets, qtcore_qobject)
- A static build of qtbase now almost works
- Fixed more path handling issues in the pro2cmake convertor script,
- Fixed dependency relationships between Main and Private modules
- Implemented proper generation for _nolink libraries (think Vulkan and OpenSSL) 
- CMake build succeeded with Coin custom instructions (yaml files) patch, will make it easier to tweak CMake build instructions, not merged yet
- More work on the configure script, it can now build with both qmake or cmake, but no command line feature mapping yet

11. April 2019

- qtbase can now be built on Windows 10 in a private Coin instance,
- Research ongoing on building qtsvg against an installed qtbase, some issues already discovered
- Research into cross compiling to iOS by leveraging latest CMake 3.14 iOS support (which is still rudimentary)
- Investigated Conan, hunter and vcpkg in regards to providing 3rd party packages for iOS
- Ongoing work on providing packages and targets for module tools (CoreTools -> moc, rcc, WidgetTools -> uic, etc) which can then be used during cross building
- Fixed pro2cmake generation script to handle .pri includes and files better (fewer things that are not found),
- Added support to the generation script for qmake style no_link libraries (openssl, vulkan),
- Continuing work on merging qt5 dev branch to wip/cmake 
- Worked on infrastructure in Coin to allow building projects both with qmake and cmake (or anything else really), a prototype is available but still needs some work
- Continuing work on the configure script, and the mapping between configure features and cmake features
- Discussion ongoing on how to handle building and running tests in Coin

04. April 2019

- Work done on using a yocto generated sysroot to cross-build qtbase to an embedded device (some qtbase patches pending review, some patches still need to be upstreamed to the meta-b2qt repo)
- Regenerated many CMakeLists.txt files with the pro2cmake script
- WIP fixing the conversion script to handle .pri inclusion paths better
- WIP fixing the Windows provisioning to build the Qt CMake port on Windows
- Tested building of qtbase on macOS and Linux on a private instance of Coin (succeeds)
- WIP fixing the conversion script to deal with qmake's "SUBDIRS -=" functionality, which is not supported natively by CMake
- Progress on changing the qtbase configure script to use CMake to build Qt (rudimentary support, no feature mapping yet)
- Various small fixes to omissions in regenerated CMakeLists.txt files

21. March 2019

- Improved pro2cmake script to regenerate Core, Network, Gui, Widgets modules, to have smaller diffs in anticipation of a dev -> wip/cmake merge
- Fixed windows build (qpa plugin, various small conditions and other tiny things)
- Fixed automoc and autorcc not to hang due to incorrect target dependencies (upstream patch)
- Fixed certain incorrect target dependencies
- Improved pro2cmake script to parse which files need to be processed by rcc, instead of using rcc for that
- Worked on provisioning of latest CMake binaries on macOS and Linux
- Worked on getting private instances of Coin to build the CMake port on macOS and Linux
- Initial work on getting the configure script to invoke CMake instead of qmake for building qtbase