Use code coverage to detect which tests to run


#1

Just read through Gregory Szorc’s notes on Facebook’s Dev Infrastructure at Scale talk and wanted to share:

http://gregoryszorc.com/blog/2015/03/28/notes-from-facebook's-developer-infrastructure-at-scale-f8-talk/

In particular the section on how they do CI seems worth reading. One neat thing they do to speed up the process is to use code coverage to determine which tests to run.

Thought it would be worth discussing a bit to see if we could do something similar with velocity:core and what would be involved.

EDIT: Just saw that the HN thread for the main link discusses this too. This sounds like a nice, simple approach:

We have a home grown solution for this. We use pants (https://pantsbuild.github.io/) for our Python builds and phabricator for code reviews. Pants gives us a dependency graph so from which files get touched in a given diff, we can feed that back into pants to get the targets to build and the test suites to run. It’s not as granular as running individual tests in a suite for small changes, but it definitely helps cut most runs down to just a small percentage of all the tests. The ability to pants to build test suites in isolation is also pretty key since it means we can run a whole bunch of suites in parallel so in many cases the total test run time can be just the time of the longest individual suite.
As others have mentioned, kicking off full runs on a regular basis is important to make sure there aren’t weird cases that break and aren’t caught in this approach." - andrewjshults


#2

I’ve thought about this and think I have identified the steps needed to make it work.

I already wrote a coverage tool for Velocity and I’m hoping to get that working again in the next couple of weeks. The coverage tool is what we need first, then we’ll have the info needed to know which test files affect tests thus allowing us to rerun only relevant tests. This will be huge for slow running tests like e2e. Add parallelization and maybe we can truly have e2e tests running on save!


#3

https://github.com/xolvio/meteor-coverage this is the code if you’re interested in looking