|
|
| Line 1: |
Line 1: |
| This session is about quality assuring the performance of Qt, with a special emphasis on preventing performance regressions.
| |
|
| |
|
| First, we identify fundamental challenges that any system that attempts to detect (and ultimately minimize the occurrence of) performance regressions in Qt is likely to have to deal with.
| |
|
| |
| Second, a couple of practical ways to facilitate contributions to this effort are proposed.
| |
|
| |
| The goal of the session is to get feedback and collect additional ideas.
| |
|
| |
| ————————————————-
| |
|
| |
| =Background / proposal=
| |
|
| |
| * Scope: Preventing performance regressions
| |
| * Main requirement: Report a performance regression ''if and only if'' there really is one in Qt
| |
| ** Avoid false positives/negatives
| |
| * Challenges:
| |
| ** Coverage (what parts of Qt are tested for performance?)
| |
| ** Stability (of test environment or test itself)
| |
| ** Precision (ideally each commit should be tested, but it would require more resources)
| |
| ** Exceptions (sometimes performance needs to be sacrificed)
| |
| * Possible contributions:
| |
| ** Submit a patch to Qt that ''does not worsen'' performance
| |
| ** Submit a patch to Qt that ''improves'' performance (e.g. by fixing an existing regression)
| |
| ** Submit a new performance test to increase the coverage
| |
| * Possible facilitations:
| |
| ** Provide access to QA reports:
| |
| *** Known performance regressions
| |
| *** Current performance test coverage
| |
| ** Provide feedback to commits:
| |
| *** “Sorry, your commit introduced a performance regression …”
| |
| *** “Congratulations! Your commit improved performance …”
| |
|
| |
| =Comments / discussion=
| |
|
| |
| * How do we ensure that people submit performance benchmark in Open Governance?
| |
| * Counting instruction reads may not be a useful metric in some cases (e.g. apps involving spinlocks).
| |
| * Start-up time of an application is an important aspect of performance.
| |
| * There was a promise around Qt4 that Qt start-up time would be reduced, which was not kept. Is start-up time verified during development?
| |
| * Perceived performance is very different from real performance. How can Qt help facilitate improving the perceived quality?
| |
| * Use git to pin-point the introduction of a performance regression (bisect)
| |
| * Coverage: For performance, this is tricky. Function coverage not reliable since performance can vary with different parameters. 3% coverage could be perfectly acceptable for performance benchmark tests, if the right 3% have been covered.
| |
| * Could Qt allow performance profiles for different platforms, that allows Qt behave differently depending on the platform it runs on?
| |
| * In order to find out where performance testing should be focused, we should measure usage statistics of Qt being run, to see where Qt uses most of it’s time during execution.
| |
| * Improving performance in a part of Qt isn’t always good as it could introduce undesirable side-effects.
| |
| * Likewise, sacrificing performance in a part of Qt is sometimes necessary.
| |