Qt Metrics Page Backlog

Qt Metrics Page Backlog

This is the development backlog for Qt Metrics page: http://testresults.qt.io/qtmetrics/ [testresults.qt.io]

For more detailed description of the Qt Metrics page please see the wiki page: http://wiki.qt.io/Qt_Metrics_Page_Description [qt.io]

In case of proposals or improvement ideas please contact the development team:
Juha Sippola: report builder (qt.io/member/137658 [qt.io])
Tony Sarajärvi: parser and database (qt.io/member/132527 [qt.io])

See the DONE items after v1.0 at the bottom of the page.

The Backlog

STATUS	VERSION	PRIORITY	TAG	PAGE / BOX	ITEM	CODE REVIEW
idle		***	Functionality	CI / Graphs	CI graphs to visualize CI performance: a) autotest set execution time (to visualize sequential and parallel runs), b) the trend of phase duration (one, or several if feasible) across the last 20 builds in selected Project Configuration. Draft indicators drawn for both of these. Utilize the data already available in the database.
idle		***	Functionality	Common / Graphs	Create improvement progress trend graphs, e.g. on build success rates, build execution times, Configuration and Autotest force success and insignificant flag removals.
idle		**	Functionality	RTA	Certain RTA data jamms the Qt RTA Metrics tool. Corrupted data available for study.
idle		**	Usability	Common	Enable direct URL usage by filtering via URL parameters (e.g. to mail direct link to a specific report). This is a known problem in Ajax implementations.
idle		**	Usability	Common	Browser Back button does not work with nested metrics (you must use the top links in a box instead). This is related to the direct URL usage item.
idle		**	Look’n‘feel	CI	The build history information in the dashboards is currently shown in a table format. A passed/failed % graphics solution could provide additional readability and improved visual appearance, both in the current build histories and in new trend graphs where needed.
idle		**	Functionality	New	Report generator to create a detailed text & graph report from a view. This would be applicable for such data that takes long time to extract and format. The content of the reports must be specified first.
idle		**	Functionality	CI / Project dashboard	Modify the list of projects on level 1 to read only selected Configuration data. Currently the Configuration filter affects only to what projects are listed, not to the data that is shown for the listed projects (the data is from Project level). This is an improvement to the v2.6.
idle		**	Functionality	CI / Project dashboard	Include a mechanism to sort the lists by title (like in Autotest dashboard)
idle		**	Functionality	CI / Autotest dashboard	Have a mechanism to be able to see what went wrong with the autotests, and then assigning a corrective task to the right person to fix it. This needs to be specified in more details (test case metrics provide a partial solution already).
idle		**	Functionality	New page	Add a new metric/page for the build phases data (which CI build machines failed most often and which CI builds and phases take the longest time to execute). This is already supported in the parser and the data is stored into the database. The metric report needs to be specified first. The graph in Project dashboard already touches this a bit.
idle		**	Functionality	CI / Autotest dashboard	Include the bug id’s for test cases. This requires first a database or other solution where to get the data.
idle		**	Usability	Common	Include the mostly needed links to header e.g. as a tag cloud or similar, for example with items like “to whom”, “what”, “how”, “examples”, “support”, “contribute”. This would improve the overall user experience.
idle		**	Usability	CI / Project dashboard	Have the number of shown builds configurable in the build history, instead of the current fixed amount.
idle		**	Implementation	Common	Add Google Analytics to get information on the portal usage.
idle		*	Functionality	Common / Graphs	Send graphs by email, or register to a list to receive certain graphs at certain intervals.
idle		*	Functionality	CI / Project dashboard	Tick box to select if idle times are included or not in the Build phases graph. This is an improvement to the v2.8.
idle		*	Functionality	CI / Autotest dashboard	The number (and list) of projects where autotest (set) is run could be included in the Autotest level 1.
idle		*	Functionality	CI / Autotest dashboard	Have the test case passed/failed/skipped summary in autotest dashboard on level 2 (could be calculated e.g. on the level 1 where already read from db and saved to session vars); v2.7 already lists data into build history table as pop-ups but some kind of summary of latest/selected build could be added. This is an improvement to the v2.7.
idle		*	Look’n’feel	CI / Autotest dashboard	‘Official’ icons for info and sorting (see pics directory), and maybe the build log file links in autotest dashboard on level 2 (e.g. from iconmonstr.com).
idle		*	Security	Common	Qt metrics page cannot be accessed via https currently
idle		*	Usability	Filters	Have a ‘clear the filter’ link/icon in each displayed filter in dashboard filter areas (so that you don’t have to select All from the filter box)
idle		*	Usability	Filters	Indicate somehow that some filter is active and Clear selections ‘can’ be pressed
idle		*	Usability	CI / Project dashboard	Project dashboard to include number of significant and insignificant vs. total autotests in latest Build -> both in 1st (Project list) and 2nd (Conf list) level
idle		*	Usability	CI / Project dashboard	Configuration result to the Build phases graph (e.g. to conf title). This is an improvement to the v2.8.
idle		*	Usability	CI / Project dashboard	Add tooltips into Build phases graph (e.g phase name and value). This is an improvement to the v2.8.
idle		*	Usability	CI / Project dashboard	The Project Build history and Configurations show ‘latest build’ for the previously latest one if there is a new build created during the browsing. User need to refresh to get this in synch again. This is an improvement to the v2.3.
idle		*	Usability	CI / Autotest dashboard	Show expected duration to load the xml data in the Autotest dashboard level 2
idle		*	Usability	CI	Check log file availability (it may not always be available)
idle		*	Usability	CI	Error handling for cases where e.g. Project, Conf or Autotest drops from the database (due to database rebuild); unsure how frequently these kind of situations would happen
idle		*	Implementation	Common / Graphs	Allow multiple graph scripts per metrics box (add loop to metricsboxci.php). This is an improvement to the v2.8.

Done Items

Changes after v1.0:

STATUS	VERSION	PRIORITY	TAG	PAGE / BOX	ITEM	CODE REVIEW
DONE	v2.8	***	Functionality	CI	Build phases graph to visualize CI performance (project build phases by configurations). Show how long do the builds take to execute, and are the different configurations run effectively in parallel. Utilize the data already available in the database (phases table).	link [codereview.qt.io]
DONE	v2.7	***	Functionality	Autotest dashboard	Autotest case details: Include the number of reruns in the autotest views. By analyzing the so-called flaky tests, those autotests or test cases that first fail but then pass on second round without any changes, could help to decrease the overall build times. This is already supported in the parser and the database implementation. Purpose is to automate the flaky tests part in the currently manually generated report [qt.io]. In addition, the autotest result history colors (autotest dashboard level 2) could be made more precise with the new data in the database.	link [codereview.qt.io]
DONE	v2.6	***	Filters	CI	Group Configuration filter values to a Platform filter (the current filters and the links in the dashboards provide the same functionality and are therefore unnecessary duplicate to each other). In addition, when only Configuration (or Platform) filtered, show the Project list for the Conf (or Platform) instead of the current note to select a Project in the Project dashboard.	link [codereview.qt.io]
DONE	v2.5	***	Filters	CI	Split Project filter to project name (easier filtering for project owners) and to Branch (dev,release,stable) to filter e.g. Project and Autotest list at the 1st level (the current filters and the links in the dashboards provide the same functionality and are therefore unnecessary duplicate to each other).	link [codereview.qt.io]
DONE	v2.4	***	Usability / Performance	Autotest dashboard	Autotest Failed % calculation performance: The data on level 1 when Project is filtered takes time to calculate. Replace the current approach of reading from the XML files with reading from database. In addition, Failed % on level 1 could be shown also when Project is not filtered to have an overall view to the most failing autotests like originally requested. New database table(s) are needed with changes to the parser.	link [codereview.qt.io]
DONE	v2.3	***	Usability	CI / Project dashboard	Project dashboard for any build i.e. to show the data (general, builds etc.) for any build (by clicking e.g. the build number); now the shows only the latest build	link [codereview.qt.io]
DONE	v2.2	****	Functionality	Autotest dashboard	Autotest case results: Detail the results of the autotest sets into test case level, which would make it easier to point out the detailed failure reasons and therefore to get the corrections faster as well. Purpose is to automate the currently manually generated report [qt.io].	link [codereview.qt.io]
DONE	v2.2	***	Usability	CI / Autotest dashboard	Autotest dashboard to include number of executed autotests (passed/failed). Specially after per-step precision is implemented.	link [codereview.qt.io]
DONE	v2.1	****	Usability / Performance	CI	Performance improvements to display the requested information in reasonable time after request (in about 5 to 7 seconds instead of several minutes in the worst case). Currently identified problems are a) initial page loading (reading complete data and saving it into browser session variables), b) page loading during a database rebuild, and c) first read operations after a database update. Possible causes are the speed of the testresults server, speed of the database, rebuild of the DBMS cache after each update and the report builder application logic. Solutions include modifying the database server and its configurations, plus the database parameters and structure, adding database indexing, and improving the report builder application logic.	link [codereview.qt.io]
DONE	v2.0	***	Functionality	Common	Implement the main menu support for several metrics pages and to identify active and inactive menu items	link [codereview.qt.io]
DONE	v2.0	***	Security	Common	Store the database login information to a secured file (although this is currently not a problem because the user has only read access to the database, and the login is restricted by the server)	link [codereview.qt.io]
DONE	v1.5.3	***	Implementation	Common	Ajax and metrics box structure refactoring to make it easier to add new metrics pages and to avoid duplicate code.	link [codereview.qt.io]
DONE	v1.5.1, v1.5	**	Implementation	CI / Project dashboard	Solve the URL encoding issue with correct PHP/JavaScript functions instead of the current string manipulation	link [codereview.qt.io], link [codereview.qt.io]
DONE	–	***	Usability	Common	Automatic frequent database updates from the build log files. See also here [bugreports.qt.io]. Note the possible negative affects to the performance.	link [codereview.qt.io]
DONE	v1.4	***	Usability	CI / Autotest dashboard	Sorting based on any column in the list (in level 1 view)	link [codereview.qt.io]
DONE	v1.3	***	Functionality	CI / Autotest dashboard	Add result history for Project/Conf builds	link [codereview.qt.io]
DONE	v1.2	****	Filters	CI	Add timescale filter (e.g. since a date) and apply it to Project dashboard and Autotest dashboard	link [codereview.qt.io]
DONE	v1.1.1	***	Usability	CI	Change the order in which the boxes are loaded on initial page load to indicate the possible database rebuild progress instead of welcome text when loading the page during rebuild	link [codereview.qt.io]
DONE	v1.1	****	Usability	CI	Show database status (last update time); compare session start time to database rebuild time and show rebuild progress (if ongoing) or reload proposal text (when done) to the user; status is updated when surfing through the filters	link [codereview.qt.io]