Qt Metrics Page Backlog

h1. Qt Metrics Page Backlog

This is the development backlog for Qt Metrics page: "http://testresults.qt.io/qtmetrics/":http://testresults.qt.io/qtmetrics/

For more detailed description of the Qt Metrics page please see the wiki page: "http://wiki.qt.io/Qt_Metrics_Page_Description":http://wiki.qt.io/Qt_Metrics_Page_Description

In case of proposals or improvement ideas please contact the development team:
Juha Sippola: report builder ("qt.io/member/137658":https://qt.io/member/137658)
Tony Sarajärvi: parser and database ("qt.io/member/132527":https://qt.io/member/132527)

See the DONE items after v1.0 at the bottom of the page.

The Backlog

{background:#009900}. |. STATUS |. VERSION |. PRIORITY |. TAG |. PAGE / BOX |. ITEM |_. CODE REVIEW |
| idle | | * | Functionality | CI / Graphs | CI graphs to visualize CI performance: a) autotest set execution time (to visualize sequential and parallel runs), b) the trend of phase duration (one, or several if feasible) across the last 20 builds in selected Project Configuration. Draft indicators drawn for both of these. Utilize the data already available in the database. | |
| idle | |'* | Functionality | Common / Graphs | Create improvement progress trend graphs, e.g. on build success rates, build execution times, Configuration and Autotest force success and insignificant flag removals. | |
| idle | | ' | Functionality | RTA | Certain RTA data jamms the Qt RTA Metrics tool. Corrupted data available for study. | |
| idle | | ' | Usability | Common | Enable direct URL usage by filtering via URL parameters (e.g. to mail direct link to a specific report). This is a known problem in Ajax implementations. | |
| idle | | ' | Usability | Common | Browser Back button does not work with nested metrics (you must use the top links in a box instead). This is related to the direct URL usage item. | |
| idle | | ' | Look'n'feel | CI | The build history information in the dashboards is currently shown in a table format. A passed/failed % graphics solution could provide additional readability and improved visual appearance, both in the current build histories and in new trend graphs where needed. | |
| idle | | ' | Functionality | New | Report generator to create a detailed text & graph report from a view. This would be applicable for such data that takes long time to extract and format. The content of the reports must be specified first. | |
| idle | | ' | Functionality | CI / Project dashboard | Modify the list of projects on level 1 to read only selected Configuration data. Currently the Configuration filter affects only to what projects are listed, not to the data that is shown for the listed projects (the data is from Project level). This is an improvement to the v2.6. | |
| idle | | ' | Functionality | CI / Project dashboard | Include a mechanism to sort the lists by title (like in Autotest dashboard) | |
| idle | | ' | Functionality | CI / Autotest dashboard | Have a mechanism to be able to see what went wrong with the autotests, and then assigning a corrective task to the right person to fix it. This needs to be specified in more details (test case metrics provide a partial solution already). | |
| idle | | ' | Functionality | New page | Add a new metric/page for the build phases data (which CI build machines failed most often and which CI builds and phases take the longest time to execute). This is already supported in the parser and the data is stored into the database. The metric report needs to be specified first. The graph in Project dashboard already touches this a bit. | |
| idle | | ' | Functionality | CI / Autotest dashboard | Include the bug id’s for test cases. This requires first a database or other solution where to get the data. | |
| idle | | ' | Usability | Common | Include the mostly needed links to header e.g. as a tag cloud or similar, for example with items like "to whom", "what", "how", "examples", "support", "contribute". This would improve the overall user experience. | |
| idle | | ' | Usability | CI / Project dashboard | Have the number of shown builds configurable in the build history, instead of the current fixed amount. | |
| idle | | ' | Implementation | Common | Add Google Analytics to get information on the portal usage. | |
| idle | | * | Functionality | Common / Graphs | Send graphs by email, or register to a list to receive certain graphs at certain intervals. | |
| idle | | * | Functionality | CI / Project dashboard | Tick box to select if idle times are included or not in the Build phases graph. This is an improvement to the v2.8. | |
| idle | | * | Functionality | CI / Autotest dashboard | The number (and list) of projects where autotest (set) is run could be included in the Autotest level 1. | |
| idle | | * | Functionality | CI / Autotest dashboard | Have the test case passed/failed/skipped summary in autotest dashboard on level 2 (could be calculated e.g. on the level 1 where already read from db and saved to session vars); v2.7 already lists data into build history table as pop-ups but some kind of summary of latest/selected build could be added. This is an improvement to the v2.7. | |
| idle | | * | Look’n’feel | CI / Autotest dashboard | ‘Official’ icons for info and sorting (see pics directory), and maybe the build log file links in autotest dashboard on level 2 (e.g. from iconmonstr.com). | |
| idle | | * | Security | Common | Qt metrics page cannot be accessed via https currently | |
| idle | | * | Usability | Filters | Have a ‘clear the filter’ link/icon in each displayed filter in dashboard filter areas (so that you don't have to select All from the filter box) | |
| idle | | * | Usability | Filters | Indicate somehow that some filter is active and Clear selections ‘can’ be pressed | |
| idle | | * | Usability | CI / Project dashboard | Project dashboard to include number of significant and insignificant vs. total autotests in latest Build > both in 1st (Project list) and 2nd (Conf list) level | |
| idle | | * | Usability | CI / Project dashboard | Configuration result to the Build phases graph (e.g. to conf title). This is an improvement to the v2.8. | |
| idle | | * | Usability | CI / Project dashboard | Add tooltips into Build phases graph (e.g phase name and value). This is an improvement to the v2.8. | |
| idle | | * | Usability | CI / Project dashboard | The Project Build history and Configurations show ‘latest build’ for the previously latest one if there is a new build created during the browsing. User need to refresh to get this in synch again. This is an improvement to the v2.3. | |
| idle | | * | Usability | CI / Autotest dashboard | Show expected duration to load the xml data in the Autotest dashboard level 2 | |
| idle | | * | Usability | CI | Check log file availability (it may not always be available) | |
| idle | | * | Usability | CI | Error handling for cases where e.g. Project, Conf or Autotest drops from the database (due to database rebuild); unsure how frequently these kind of situations would happen | |
| idle | | * | Implementation | Common / Graphs | Allow multiple graph scripts per metrics box (add loop to metricsboxci.php). This is an improvement to the v2.8. | |

h2. Done Items
Changes after v1.0:
{background:#009900}. |. STATUS |. VERSION |. PRIORITY |. TAG |. PAGE / BOX |. ITEM |_. CODE REVIEW |
| DONE | v2.8 | * | Functionality | CI | Build phases graph to visualize CI performance (project build phases by configurations). Show how long do the builds take to execute, and are the different configurations run effectively in parallel. Utilize the data already available in the database (phases table). | "link":https://codereview.qt.io/88722 |
| DONE | v2.7 |* | Functionality | Autotest dashboard | Autotest case details: Include the number of reruns in the autotest views. By analyzing the so-called flaky tests, those autotests or test cases that first fail but then pass on second round without any changes, could help to decrease the overall build times. This is already supported in the parser and the database implementation. Purpose is to automate the flaky tests part in the currently "manually generated report":http://wiki.qt.io/CI_Failing_Tests. In addition, the autotest result history colors (autotest dashboard level 2) could be made more precise with the new data in the database. | "link":https://codereview.qt.io/88271 |
| DONE | v2.6 | * | Filters | CI | Group Configuration filter values to a Platform filter (the current filters and the links in the dashboards provide the same functionality and are therefore unnecessary duplicate to each other). In addition, when only Configuration (or Platform) filtered, show the Project list for the Conf (or Platform) instead of the current note to select a Project in the Project dashboard. | "link":https://codereview.qt.io/88052 |
| DONE | v2.5 |* | Filters | CI | Split Project filter to project name (easier filtering for project owners) and to Branch (dev,release,stable) to filter e.g. Project and Autotest list at the 1st level (the current filters and the links in the dashboards provide the same functionality and are therefore unnecessary duplicate to each other). | "link":https://codereview.qt.io/87688 |
| DONE | v2.4 | * | Usability / Performance | Autotest dashboard | Autotest Failed % calculation performance: The data on level 1 when Project is filtered takes time to calculate. Replace the current approach of reading from the XML files with reading from database. In addition, Failed % on level 1 could be shown also when Project is not filtered to have an overall view to the most failing autotests like originally requested. New database table(s) are needed with changes to the parser. | "link":https://codereview.qt.io/87479 |
| DONE | v2.3 |* | Usability | CI / Project dashboard | Project dashboard for any build i.e. to show the data (general, builds etc.) for any build (by clicking e.g. the build number); now the shows only the latest build | "link":https://codereview.qt.io/87102 |
| DONE | v2.2 | ' | Functionality | Autotest dashboard | Autotest case results: Detail the results of the autotest sets into test case level, which would make it easier to point out the detailed failure reasons and therefore to get the corrections faster as well. Purpose is to automate the currently "manually generated report":http://wiki.qt.io/CI_Failing_Tests. | "link":https://codereview.qt.io/86345 |
| DONE | v2.2 | * | Usability | CI / Autotest dashboard | Autotest dashboard to include number of executed autotests (passed/failed). Specially after per-step precision is implemented. | "link":https://codereview.qt.io/86345 |
| DONE | v2.1 | | Usability / Performance | CI | Performance improvements to display the requested information in reasonable time after request (in about 5 to 7 seconds instead of several minutes in the worst case). Currently identified problems are a) initial page loading (reading complete data and saving it into browser session variables), b) page loading during a database rebuild, and c) first read operations after a database update. Possible causes are the speed of the testresults server, speed of the database, rebuild of the DBMS cache after each update and the report builder application logic. Solutions include modifying the database server and its configurations, plus the database parameters and structure, adding database indexing, and improving the report builder application logic. | "link":https://codereview.qt.io/81924 |
| DONE | v2.0 | * | Functionality | Common | Implement the main menu support for several metrics pages and to identify active and inactive menu items | "link":https://codereview.qt.io/76861 |
| DONE | v2.0 |* | Security | Common | Store the database login information to a secured file (although this is currently not a problem because the user has only read access to the database, and the login is restricted by the server) | "link":https://codereview.qt.io/76861 |
| DONE | v1.5.3 | * | Implementation | Common | Ajax and metrics box structure refactoring to make it easier to add new metrics pages and to avoid duplicate code. | "link":https://codereview.qt.io/74962 |
| DONE | v1.5.1, v1.5 | | Implementation | CI / Project dashboard | Solve the URL encoding issue with correct PHP/JavaScript functions instead of the current string manipulation | "link":https://codereview.qt.io/74146, "link":https://codereview.qt.io/65575 |
| DONE | | * | Usability | Common | Automatic frequent database updates from the build log files. See also "here":https://bugreports.qt.io/browse/QTQAINFRA-639. Note the possible negative affects to the performance. | "link":https://codereview.qt.io/62764 |
| DONE | v1.4 |* | Usability | CI / Autotest dashboard | Sorting based on any column in the list (in level 1 view) | "link":https://codereview.qt.io/59953 |
| DONE | v1.3 | * | Functionality | CI / Autotest dashboard | Add result history for Project/Conf builds | "link":https://codereview.qt.io/59916 |
| DONE | v1.2 | | Filters | CI | Add timescale filter (e.g. since a date) and apply it to Project dashboard and Autotest dashboard | "link":https://codereview.qt.io/59736 |
| DONE | v1.1.1 | '* | Usability | CI | Change the order in which the boxes are loaded on initial page load to indicate the possible database rebuild progress instead of welcome text when loading the page during rebuild | "link":https://codereview.qt.io/58499 |