CMake Port/Development Notes

2019

23. May 2019

- Coin now tests every single patch pushed to Gerrit to the qtbase wip/cmake branch on a single Linux Configuration, and posts the results on the Gerrit change (not enforcing), work is being done to test on Windows and macOS as well
- Most of qtimageformats has been ported
- Tests are now buildable with an installed Qt (useful for Coin)
- Implemented bundled double-conversion building (so using a package manager to install it is not strictly necessary anymore)
- Fixed hard requirement on needing Widgets when building any other module (was needed for qtimageformats)
- Fixed qmake parser to handle qmake syntax better in pro2cmake.py (doesn't crash anymore when trying to run it on qtdeclarative files )
- Continued investigation into building documentation and translation files for Qt Creator with CMake, once done the work would be forwarded to the Qt CMake Port 
- Work is being done on test data installation, so that tests that use test data can now pass

16. May 2019

- Implemented non-prefix builds (enabled with developer-build)
- Ported a part of qtimageformats
- Side work finished on porting Qt Creator to build with cmake, building docs for Qt Creator might be used for qtbase and friends
- Configure script changes merged to allow passing arbitrary cmake flags
- Continued looking into .pri file generation
- Various small fixes

09. May 2019

- Implemented script to keep manual modifications when regenerating projects
- Fixed some issues regarding DBus 
- Started looking into non-prefix builds (developer-build)
- Continued work from workshop on library mapping,
- Modified conversion script to support standalone example building
- Fixed various Linux package issues
- Regenerated a bunch of subprojects in src/
- More work on configure argument mapping
- Started looking into .pri file generation for built modules (so that qmake can use them while building apps or other qt modules)
- Continued working on the version-less targets
- Continued work on static plugins auto importing

03. May 2019 Workshop day 2

Things implemented during workshop

1. Final touches on packages providing tools
2. Final touches on keeping track which dependencies need to be found while consuming Qt in a CMake application
3. Fix current branch to be buildable with released CMake 3.14.x
4. Added a few missing xcb platform plugins
5. Implemented building qtsvg against qtbase
6. Support building qtbase & qtsvg as static libs
7. Fixed building offscreen platform plugin
8. Improved GLX detection and GL integration
9. Improved convertor script and regenerated configure.cmake files to keep track of more 3rd party dependencies
10. Updated cmake/README
11. Fixed finding host tools during cross compiling
12. Implemented exporting of major-version-less Qt targets, to facilitate easier upgrading of CMake applications from Qt5 to Qt6
13. Improved mapping of libraries from qmake to CMake land
14. Improved handling of _nolink targets where only include headers have to be propagated (Vulkan, OpenSSL)
15. Investigated the possibility of providing a user friendly Qt + CMake toolchain file for CMake applications to consume
16. Initial fixes for building qtbase with MinGW
17. Started work on building examples as standalone projects (not part of qtbase build)
18. Tested a few successful qtbase builds on a private Coin instance
19. Started work on handling static plugins in a consuming CMake application (when using a static qtbase build)

02. May 2019 Workshop day 1

Discussion points

1. Summary on the current state of CI builds of Qt-CMake-port
2. What changes have been made since last workshop
3. Be able to build applications against Qt5 or Qt6 without doing a lot of search and replaces in project files
4. What branch we want to target with our CMake changes (wip/cmake or qt5/dev or qt6/dev)
5. Time line on when we think we could merge the CMake changes into the mainline branch (Milestone 1 probably)
6. Depending on which branch we merge, decide If we need to keep qmake build of Qt working – if merged to qtbase/dev, then we need to keep qmake anyway
7. The convenience of using CMake to build applications instead of qmake (simplify what the user has to specify on the CMake command line), if we can accomplish that easily
8. What package manager we want to use for 3rd party packages (vcpkg, Conan, both, something else?)
9. What we want to do with qmake (move into different repo, keep it in qtbase?), we need the mkspec files currently which are provided in qtbase/mkspecs.
10. Configure script for CMake (do we want it? What do we want it to do?)
11. How do we want to keep up to date with qtbase/dev (regeneration of project files)
12. Examples and tests (building outside of a Qt build)

CI CMake State of the Union

Qtbase can be built currently with dirty hacks on the 3 major configs (Windows, Linux, macOS)

Implementation to allow choosing between building using CMake and QMake Qt builds properly in Coin (using yaml files in project repositories). Allows specifying build steps and options.

Tests are currently built as part of the build VM.

Timeline is 1-2 months to get the proper Coin stuff in.

What changes have been made since last workshop

We decided to get rid of qrc files, we changed instead to generate CMake rules with the pro2cmake.py convertor, which reads the qrc files and then generates a file list of files that were inside the qrc file, which are then embedded in the CMakeLists.txt. We need to do it for all modules.

TODO: Backport the qrc file conversion to qt5.

Qt has public dependencies. In static builds of Qt, private dependencies become public. We need to propagate those inside the generated QtCoreConfig.cmake files, so that when an application does find_package(Qt5Core), it also tries to find Core's dependencies. We found a way to do that in a nicer way, intead of tracking the dependencies manually (as it is done currently in KDE).

Lots and lots of improvement in the pro2cmake and configurejson2cmake convertor scripts: qrc handling, path handling, simd sources, no_link targets (vulkan, openssl), public and private dependencies handling.

As a result we merged qt5/dev into wip/cmake, and regenerated the files. This got us much closer to the current state of qtbase/dev.

Sample of special casing:

add_qt_library(Foo  
    SOURCES  
        bla.cpp foo.cpp 
    LIBRARIES  
        ZLIB::ZLIB # special case 
        # FreeType special case : remove this 
) 

We now generate Config files for each module's tools, for example Qt5CoreTools exports rcc,moc tools,

Qt5WidgetsTools exports uic tool. These packages can be used to cross-compile Qt (for Yocto).

We also were able to do a Yocto cross build.

Configure script state

It now understands –D CMake arguments and forwards them to CMake, and also qmake --foo commands, but the –foo commands are currently not converted to CMake style –D arguments, so it's just for qmake Qt builds.

It currently does not support in-source builds, and instead creates build directories and invokes CMake there.

Windows configure script has not been changed yet.

The conigure script is not strictly necessary for the CI now (which was the initial reason why we wanted configure to map to CMake options) because of the YAML changes.

It's still useful to provide the convenience for the user (so existing configure lines don't berak), so the question now is when should we do it, and what mechanism (duplicate mappings in .sh and .bat files, unified CMake script, something else?)

Be able to build applications against Qt5 or Qt6 without doing a lot of search and replaces in project files

Allow mixing Qt5 and Qt6 packages in a project, to allow partial porting of projects.

# Usecase: application wants to mix both Qt5 and Qt6, to allow gradual porting 
set(QT_CREATE_VERSIONLESS_TARGETS OFF) 
find_package(Qt5 COMPONENTS Core Gui Widgets) # Creates only Qt5::Core 
find_package(Qt6 COMPONENTS Core Gui Widgets) # Creates only Qt6::Core 

# Usecase: application doesn't mix Qt5 and Qt6, but allows to fully switch to link against either Qt5 or Qt6 
set(MY_APP_USE_QT6 TRUE) # <- set at command line by application developer 
set(MY_APP_QT_MAJOR_VERSION 6) # <- set at command line by application developer
# set(QT_CREATE_VERSIONLESS_TARGETS ON) <- Default, doesn't need to be set 
if(MY_APP_USE_QT6) 
   find_package(Qt6 COMPONENTS Core Gui Widgets) # Creates Qt6::Core, Qt::Core 
else() 
   find_package(Qt5 COMPONENTS Core Gui Widgets) # Creates Qt5::Core, Qt::Core 
endif() 

Or you could use

find_package(Qt${MY_APP_QT_MAJOR_VERSION} COMPONENTS Core Gui Widgets) # Creates Qt6::Core, Qt::Core 

What branch do we want to use for the CMake port

Make a recommendation to merge to qtbase/dev on mailing list, and make it blocking, so either CMake or qmake changes will both block integrations if build or test fails.

Pros merging into qtbase/dev

• Supporting two build system will have to be done anyway, regardless of branch, so might as well to do it in dev: because it spreads knowledge (more exposure to CMake port), that we distribute the work of maintaining the CMake changes.
• Regardless of the branch it goes into, somebody would still have to maintain two build systems, either in dev or in wip/qt6, the question is just where the changes will have to be made. If it's done in wip/qt6 -> less people can do the work (because not many people are currently involved in that branch).

Cons merging into qtbase/dev

• If a Coin Cmake config fails, it will fail the qmake one as well
• More load on the CI
• People need to learn a new build system ( but let's be real, they'll have to do it at some point anyway )

Time line on when we think we could merge the CMake changes into the mainline branch: To be discussed, but probably not before the yaml Coin change goes into production, which would take 1-2-3 months, so it depends on Coin production update.

The convenience of using CMake to build applications instead of qmake (simplify what the user has to specify on the CMake command line), if we can accomplish that easily

Provide a toolchain file, that would ease use for CMake projects, make it as easy as using qmake for a project.

This includes passing along things from configuration time like: mkspec, address sanitizer flags, original toolchain file which was used while building Qt, the public dependency Find modules (ZLIB, Atomic), build type (debug vs release) on windows, because of different MSVC runtimes.

What package manager we want to use for 3rd party packages (vcpkg, Conan, both, something else?)

Qt wants a package manager for 2 reasons:

Building 3rd party packages (to get rid of src/3rdparty)

To replace IFW with something better / nicer

The consensus is that for replacing IFW, Conan is the better choice.

For acquiring 3rd party packages, vcpkg is a much nicer solution, because Conan has some issues with package name and target name generations

(Conan generates all lower case names, while vcpkg and upstream CMake uses upper case or mixed case).

Upstream Conan issues:

https://github.com/conan-io/conan/issues/4430

https://github.com/conan-io/conan/issues/4460

https://github.com/conan-io/conan/issues/4597

https://github.com/conan-io/conan/issues/3400

What we want to do with qmake (move into different repo, keep it in qtbase?), we need the mkspec files currently which are provided in qtbase/mkspecs

Three action points:

We need to clean up the mkspecs to be integrated with CMake (generate the platform headers) (timeline is "has to happen before 6.0")

We need to make qmake compile against Qt without ifdefs (timeline is "has to happen before 6.0")

Consider moving qmake out of qtbase (timeline is later than release of qt 6.0)

Pros for keeping qmake in qtbase

Coin would test if qmake can link against QtBase within a qtbase test run, which allows us to find issues with qmake ASAP as long as we support it.

Cons

Psychological argument that qmake is still around

Coin adjustments will be needed, if qmake is in a different repo

How do we want to keep up to date with qtbase/dev (regeneration of project files)

Depend on Coin to tell us if something breaks upon a dev -> wip/cmake merge.

Proposal to make merges every two weeks, to keep up to date with new changes.

When doing a merge, do a git diff, to see if any .pro and .pri and configure.json files were touched, which is a good indicator that some CMakeLists.txt files might need to be regenerated.

Examples and tests (building outside of a Qt build)

Tests.

First workable solution would be to build tests together with Qt in Coin, and then just copy all the generated and binary files under qtbase/tests to a different machine, keeping the exact same file paths.

Examples.

Rewrite pro2cmake.py generation script to generate cmake code for examples which would be standalone and not using QtBuild.cmake functionality. This implies not using add_qt_executable, and using regular add_executable() and macros that are exported (for dbus handling, rcc handling, etc).

25. April 2019

- First merge in a while was done from qt5/dev into wip/cmake and as a result regenerated important CMakeLists.txt files.
- Fixed Windows build after qt5/dev -> wip/cmake merge and file regeneration
- Improved the exporting Tools CMake Targets patch
- Implemented 3rd party library dependency propagation when exporting module config files (packages like ZLIB or double-conversion)
- Implemented dependency propagation for a static build of Qt (targets like tinycbor, qtharfbuzz, simd targets, qtcore_qobject)
- A static build of qtbase now almost works
- Fixed more path handling issues in the pro2cmake convertor script,
- Fixed dependency relationships between Main and Private modules
- Implemented proper generation for _nolink libraries (think Vulkan and OpenSSL) 
- CMake build succeeded with Coin custom instructions (yaml files) patch, will make it easier to tweak CMake build instructions, not merged yet
- More work on the configure script, it can now build with both qmake or cmake, but no command line feature mapping yet

11. April 2019

- qtbase can now be built on Windows 10 in a private Coin instance,
- Research ongoing on building qtsvg against an installed qtbase, some issues already discovered
- Research into cross compiling to iOS by leveraging latest CMake 3.14 iOS support (which is still rudimentary)
- Investigated Conan, hunter and vcpkg in regards to providing 3rd party packages for iOS
- Ongoing work on providing packages and targets for module tools (CoreTools -> moc, rcc, WidgetTools -> uic, etc) which can then be used during cross building
- Fixed pro2cmake generation script to handle .pri includes and files better (fewer things that are not found),
- Added support to the generation script for qmake style no_link libraries (openssl, vulkan),
- Continuing work on merging qt5 dev branch to wip/cmake 
- Worked on infrastructure in Coin to allow building projects both with qmake and cmake (or anything else really), a prototype is available but still needs some work
- Continuing work on the configure script, and the mapping between configure features and cmake features
- Discussion ongoing on how to handle building and running tests in Coin

04. April 2019

- Work done on using a yocto generated sysroot to cross-build qtbase to an embedded device (some qtbase patches pending review, some patches still need to be upstreamed to the meta-b2qt repo)
- Regenerated many CMakeLists.txt files with the pro2cmake script
- WIP fixing the conversion script to handle .pri inclusion paths better
- WIP fixing the Windows provisioning to build the Qt CMake port on Windows
- Tested building of qtbase on macOS and Linux on a private instance of Coin (succeeds)
- WIP fixing the conversion script to deal with qmake's "SUBDIRS -=" functionality, which is not supported natively by CMake
- Progress on changing the qtbase configure script to use CMake to build Qt (rudimentary support, no feature mapping yet)
- Various small fixes to omissions in regenerated CMakeLists.txt files

21. March 2019

- Improved pro2cmake script to regenerate Core, Network, Gui, Widgets modules, to have smaller diffs in anticipation of a dev -> wip/cmake merge
- Fixed windows build (qpa plugin, various small conditions and other tiny things)
- Fixed automoc and autorcc not to hang due to incorrect target dependencies (upstream patch)
- Fixed certain incorrect target dependencies
- Improved pro2cmake script to parse which files need to be processed by rcc, instead of using rcc for that
- Worked on provisioning of latest CMake binaries on macOS and Linux
- Worked on getting private instances of Coin to build the CMake port on macOS and Linux
- Initial work on getting the configure script to invoke CMake instead of qmake for building qtbase