The Wikimedia Foundation shares a lot of application performance data with the world. Anyone can take a profile of a MediaWiki page load, view CPU and wall-clock flame graphs, or monitor server load in real time. This is motivated by a desire to make it easy for anyone to get involved in making Wikimedia software faster.

But there is a major obstacle to getting involved. Most performance engineering is not purely analytical but involves experimentation and trial-and-error. And while it's easy enough to inspect the performance of Wikimedia's MediaWiki stack, it is not easy to test changes. This is because Wikimedia's MediaWiki stack is increasingly complex. In order to simulate Wikimedia's production workload, it is necessary to install and configure several different software components (MediaWiki extensions, PHP extensions, caching systems, etc.) and to import a lot of content.

This should be easier. There should be a portable, reproducible performance testing set-up that anyone can turn up that approximates Wikimedia's production workload. It doesn't have to match production exactly; a rough approximation would do.

One group of developers who would likely use such a test if it were available and easy to run are PHP core developers, as a way of assessing the performance impact of changes to the language runtime itself.

@Krinkle sketched out what a representative test set-up would look like:

a "good" setup with e.g. apcu, memc, php-fpm, luasandbox, various MW extensions, and parsercache/maincache enabled. and then benchmarkParse.php with opcache.enable_cli=1 for measuring "edits" and e.g. the same Obama page behind parser cache to measure "page views" would be great as two metrics, and to then be able to iterate on that between MW releases and/or PHP releases.

Seems like a good starting point.