Run one, or a few, tests per subprocess

[sourcefrog]

Split out of #186 to be more focused on one possible solution.

Today, cargo-mutants runs cargo test which will sequentially run each test target. If any test target fails, then it stops. But if one test within a test target fails, the others will continue. Since we only care about whether any test fails, this is a waste of time, and it would be better to just stop.

One option is to run tests under Nextest (#85).

Another idea is to borrow the idea from Nextest, which is basically:

• For each test target, run it with --list to find all the contained tests
• Then run that target with --exact specifying the name of one or more tests
• After all this, run cargo test --doc to run any doctests (although maybe with an option to skip them, as they tend to be slow)

If any of those invocations fail, then we found a failure and we don't need to continue.

(However, if Nextest can be made as fast as cargo test on everything, then, rather than duplicating, perhaps we could just add features there.)

More than one test per subprocess

Nextest runs one test per subprocess, which is the obvious simple place to start.

However, spawning a subprocess has some fixed overhead (at the OS level), and also there might be globals or lazy-initialized data structures in the test code that make the first test slow to run (#85 (comment).)

So, possibly the optimal structure is to give more than one test name to the test binary, although that will mean some limited amount of wasted continuing work if the first one fails. So, maybe this should be an option that is off by default.

Parallelism

Today, the test binary will typically run several tests in parallel, controlled by --test-threads. If we tell it to run just a single test it obviously has no scope to do that.

There are at least a couple of options:

1. Give it approximately NCPUs tests to run. This would, in general, have a kind of sawtooth pattern where each task has some stragglers, so it would leave some cores idle.

2. Launch several targets in parallel. We'd expect these to be much lighter than the current jobs spawned by -j (which can each spawn many threads and processes), so it could be a separate limit. This would let us spawn a new test task as soon as a slot is available.

Extension: vary test order

This also opens an interesting opportunity to control the order in which we run tests; we could try to bias towards:

• Fast tests
• Tests whose name has something in common with the file or function being mutated
• Tests who failed before in this run
• Tests who failed on this mutation last time cargo-mutants was run, which could make incremental/repetitive runs very fast
• Any other interesting heuristic

(However this certainly doesn't have to be done in the first version.)

Timeouts

Currently timeouts apply to the whole test suite, and are calculated from the whole test suite. Running one test at a time would allow tighter timeouts, but might need a new way of calculating the timeouts. Perhaps we could start by just making the auto-timeout a multiple of the maximum time for any invocation.