We seldom know exactly why one release of GNOME is faster or slower than the last. We have various tests we can run to measure performance, but comparing results for one release with results from a previous release provides less clarity than one might hope. Even if hardware is exactly the same, there are many elements in the software stack – from the kernel, to the graphics drivers, to the toolkits, to the application itself, that can produce performance changes. From one release of GNOME to the next, each of these layers sees hundreds or thousands of changes.

It is much more useful if we can get performance numbers continuously as changes go into the codebase. We will then have a better chance of identifying a change that slows down GNOME or verifying whether if a change that was supposed to improve performance actually did. We already have a system that does continuous builds of GNOME and runs basic tests on them: GNOME Continuous. The tests that GNOME Continuous runs are, however, run on virtual machines, which poses multiple challenges for performance testing – in particular, lack of accelerated graphics, and interference from other use of the same system.

This talk will describe the new GNOME Hardware Testing initiative, which has a goal to take the builds of GNOME and base system software that GNOME Continuous creates, measure the performance on a range of performance tests, record the results, and make charts and graphs of that data available to developers. The talk will discuss technology used: how ostree is used to distribute software to the testing systems, and how upgrading, rebooting, and testing is automated to eliminate any need for manual intervention, while still allowing stock commodity hardware to be tested. It will also cover the selection of appropriate tests to reflect the performance of GNOME in a comprehensible way.

Owen Taylor

Owen Taylor has been active in the development of GNOME technologies since before GNOME existed. He maintained GTK+ and related libraries, created Pango, which enables GNOME to handle the languages of virtually all of the world’s users, and more recently led the development of GNOME Shell prior to the release of GNOME 3. His current interests relate to display technologies and the full-stack performance of GNOME. Owen is Architect for the Emerging Platform team at Red Hat, which includes the desktop and other related areas.