diff --git a/DESIGN.md b/DESIGN.md index a17875e7..7367d6e4 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -50,31 +50,57 @@ the content based on those addresses. replacements. The interface to the `syn` parser is localized here, and also the core of cargo-mutants logic to guess at valid replacements. -## Relative dependencies +## Major processing stages -After copying the tree, cargo-mutants scans the top-level `Cargo.toml` and any -`.cargo/config.toml` for relative dependencies. If there are any, the paths are -rewritten to be absolute, so that they still work when cargo is run in the -scratch directory. +1. Find the workspace enclosing the start directory, and the packages within it. +1. Determine which packages to mutate. +1. Generate mutants by walking each package. +1. Copy the source tree. +1. Run baseline tests. +1. Test each mutant in parallel. + +### Finding the workspace and packages -## Enumerating the source tree +cargo-mutants is invoked from within, or given with `-d`, a single directory, called the _start directory_. To find mutants and run tests we first need to find the enclosing workspace and the packages within it. -A source tree may be a single crate or a Cargo workspace. There are several levels of nesting: +This is done basically by parsing the output of `cargo locate-project` and `cargo metadata`. -* A workspace contains one or more packages. -* A package contains one or more targets. -* Each target names one (or possibly-more) top level source files, whose directories are walked to find more source files. -* Each source file contains some functions. -* For each function we generate some mutants. +We often want to test only one or a subset of packages in the workspace. This can be set explicitly with `--package` and `--workspace`, or heuristically depending on the project metadata and the start directory. -The name of the containing package is passed through to the `SourceFile` and the `Mutant` objects. +For each package, cargo tells us the build targets including tests and the main library or binary. The tests are not considered for mutation, so this leaves us with +some targets of interest, and for each of them cargo tells us one top source file, typically something like `src/lib.rs` or `src/main.rs`. -For source tree and baseline builds and tests, we pass Cargo `--workspace` to build and test everything. For mutant builds and tests, we pass `--package` to build and test only the package containing the mutant, on the assumption that each mutant should be caught by its own package's tests. +### Discovering mutants -Currently source files are discovered by finding any `bin` or `lib` targets in the package, then taking every `*.rs` file in the same directory as their top-level source file. (This is a bit approximate and will pick up files that might not actually be referenced by a `mod` statement, so may change in the future.) +After discovering packages and before running any tests, we discover all the potential mutants. + +Starting from the top files for each package, we parse each source file using `syn` +and then walk its AST. In the course of that walk we can find three broad categories of patterns: + +* A `mod` statement (without a block), which tells us of another source file we must remember + to walk. +* A source pattern that cargo-mutants knows how to mutate, such as a function returning a value. +* A pattern that tells cargo-mutants not to look further into this branch of the tree, such as `#[test]` or `#[mutants::skip]`. + +For baseline builds and tests, we test all the packages that will later be mutated. For mutant builds and tests, we pass `--package` to build and test only the package containing the mutant, on the assumption that each mutant should be caught by its own package's tests. We may later mutate at a granularity smaller than a single function, for example by cutting out an `if` statement or a loop, but that is not yet implemented. () +### Copying the source tree + +Mutations are tested in copies of the source tree. (An option could be added to test in-place, which would be nice for CI.) + +Initially, one copy is made to run baseline tests; if they succeed then additional copies are made as necessary for each parallel job. + +After copying the tree, cargo-mutants scans the top-level `Cargo.toml` and any +`.cargo/config.toml` for relative dependencies. If there are any, the paths are +rewritten to be absolute, so that they still work when cargo is run in the +scratch directory. + +Currently, the whole workspace tree is copied. In future, possibly only the package to be mutated could be copied: this would require changes to the code that fixes up dependencies. + +(This current approach assumes that all the packages are under the workspace directory, which is common but not actually required.) + ## Handling timeouts Mutations can cause a program to go into an infinite (or just very long) loop: diff --git a/NEWS.md b/NEWS.md index 3bfe214f..5b6836be 100644 --- a/NEWS.md +++ b/NEWS.md @@ -2,6 +2,8 @@ ## Unreleased +- Changed: `cargo mutants` now tries to match the behavior of `cargo test` when run within a workspace. If run in a package directory, it tests only that package. If run in a workspace that is not a package (a "virtual workspace"), it tests the configured default packages, or otherwise all packages. This can all be overridden with the `--package` or `--workspace` options. + - New: generate key-value map values from types like `BTreeMap>`. - Changed: Send trace messages to stderr rather stdout, in part so that it won't pollute json output. diff --git a/book/src/workspaces.md b/book/src/workspaces.md index 2afb4bcc..b944093f 100644 --- a/book/src/workspaces.md +++ b/book/src/workspaces.md @@ -2,17 +2,20 @@ cargo-mutants supports testing Cargo workspaces that contain multiple packages. The entire workspace tree is copied. -By default, all source files in all packages in the workspace are tested. +By default, cargo-mutants has [the same behavior as Cargo](https://doc.rust-lang.org/cargo/reference/workspaces.html): -**NOTE: This behavior is likely to change in future: see .** - -You can use the `--file` options to restrict cargo-mutants to testing only files -from some subdirectory, e.g. with `-f "utils/**/*.rs"`. (Remember to quote globs -on the command line, so that the shell doesn't expand them.) You can use `--list` or -`--list-files` to preview the effect of filters. +* If `--workspace` is given, all packages in the workspace are tested. +* If `--package` is given, the named packages are tested. +* If the starting directory (or `-d` directory) is in a package, that package is tested. +* Otherwise, the starting directory must be in a virtual workspace. If it specifies default members, they are tested. Otherwise, all packages are tested. For each mutant, only the containing package's tests are run, on the theory that each package's tests are responsible for testing the package's code. The baseline tests exercise all and only the packages for which mutants will be generated. + +You can also use the `--file` options to restrict cargo-mutants to testing only files +from some subdirectory, e.g. with `-f "utils/**/*.rs"`. (Remember to quote globs +on the command line, so that the shell doesn't expand them.) You can use `--list` or +`--list-files` to preview the effect of filters. diff --git a/src/build_dir.rs b/src/build_dir.rs index 26038ed9..76573aca 100644 --- a/src/build_dir.rs +++ b/src/build_dir.rs @@ -152,10 +152,8 @@ mod test { #[test] fn build_dir_debug_form() { let options = Options::default(); - let root = CargoTool::new() - .find_root("testdata/tree/factorial".into()) - .unwrap(); - let build_dir = BuildDir::new(&root, &options, &Console::new()).unwrap(); + let workspace = Workspace::open("testdata/tree/factorial").unwrap(); + let build_dir = BuildDir::new(&workspace.dir, &options, &Console::new()).unwrap(); let debug_form = format!("{build_dir:?}"); assert!( Regex::new(r#"^BuildDir \{ path: "[^"]*[/\\]cargo-mutants-factorial[^"]*" \}$"#) diff --git a/src/cargo.rs b/src/cargo.rs index 58d74783..b4ee142b 100644 --- a/src/cargo.rs +++ b/src/cargo.rs @@ -3,148 +3,51 @@ //! Run Cargo as a subprocess, including timeouts and propagating signals. use std::env; -use std::sync::Arc; +use std::time::{Duration, Instant}; -use anyhow::{anyhow, ensure, Context, Result}; -use camino::{Utf8Path, Utf8PathBuf}; +use anyhow::Result; +use camino::Utf8Path; use itertools::Itertools; -use serde_json::Value; -use tracing::debug_span; -#[allow(unused_imports)] -use tracing::{debug, error, info, span, trace, warn, Level}; +use tracing::{debug, debug_span}; -use crate::process::get_command_output; -use crate::source::Package; -use crate::tool::Tool; +use crate::outcome::PhaseResult; +use crate::package::Package; +use crate::process::Process; use crate::*; -#[derive(Debug)] -pub struct CargoTool { - // environment is currently constant across all invocations. - env: Vec<(String, String)>, -} - -impl CargoTool { - pub fn new() -> CargoTool { - let env = vec![ - ("CARGO_ENCODED_RUSTFLAGS".to_owned(), rustflags()), - // The tests might use Insta , and we don't want it to write - // updates to the source tree, and we *certainly* don't want it to write - // updates and then let the test pass. - ("INSTA_UPDATE".to_owned(), "no".to_owned()), - ]; - CargoTool { env } - } -} - -impl Tool for CargoTool { - fn name(&self) -> &str { - "cargo" - } - - fn find_root(&self, path: &Utf8Path) -> Result { - ensure!(path.is_dir(), "{path:?} is not a directory"); - let cargo_bin = cargo_bin(); // needed for lifetime - let argv: Vec<&str> = vec![&cargo_bin, "locate-project", "--workspace"]; - let stdout = get_command_output(&argv, path) - .with_context(|| format!("run cargo locate-project in {path:?}"))?; - let val: Value = - serde_json::from_str(&stdout).context("parse cargo locate-project output")?; - let cargo_toml_path: Utf8PathBuf = val["root"] - .as_str() - .with_context(|| format!("cargo locate-project output has no root: {stdout:?}"))? - .to_owned() - .into(); - debug!(?cargo_toml_path, "Found workspace root manifest"); - ensure!( - cargo_toml_path.is_file(), - "cargo locate-project root {cargo_toml_path:?} is not a file" - ); - let root = cargo_toml_path - .parent() - .ok_or_else(|| anyhow!("cargo locate-project root {cargo_toml_path:?} has no parent"))? - .to_owned(); - ensure!( - root.is_dir(), - "apparent project root directory {root:?} is not a directory" - ); - Ok(root) - } - - /// Find the root files for each relevant package in the source tree. - /// - /// A source tree might include multiple packages (e.g. in a Cargo workspace), - /// and each package might have multiple targets (e.g. a bin and lib). Test targets - /// are excluded here: we run them, but we don't mutate them. - /// - /// Each target has one root file, typically but not necessarily called `src/lib.rs` - /// or `src/main.rs`. This function returns a list of all those files. - /// - /// After this, there is one more level of discovery, by walking those root files - /// to find `mod` statements, and then recursively walking those files to find - /// all source files. - fn top_source_files(&self, source_root_path: &Utf8Path) -> Result>> { - let cargo_toml_path = source_root_path.join("Cargo.toml"); - debug!(?cargo_toml_path, ?source_root_path, "Find root files"); - check_interrupted()?; - let metadata = cargo_metadata::MetadataCommand::new() - .manifest_path(&cargo_toml_path) - .exec() - .context("run cargo metadata")?; - - let mut r = Vec::new(); - // cargo-metadata output is not obviously ordered so make it deterministic. - for package_metadata in metadata - .workspace_packages() - .iter() - .sorted_by_key(|p| &p.name) - { - check_interrupted()?; - let _span = debug_span!("package", name = %package_metadata.name).entered(); - let manifest_path = &package_metadata.manifest_path; - debug!(%manifest_path, "walk package"); - let relative_manifest_path = manifest_path - .strip_prefix(source_root_path) - .map_err(|_| { - anyhow!( - "manifest path {manifest_path:?} for package {name:?} is not within the detected source root path {source_root_path:?}", - name = package_metadata.name - ) - })? - .to_owned(); - let package = Arc::new(Package { - name: package_metadata.name.clone(), - relative_manifest_path, - }); - for source_path in direct_package_sources(source_root_path, package_metadata)? { - check_interrupted()?; - r.push(Arc::new(SourceFile::new( - source_root_path, - source_path, - &package, - )?)); - } - } - Ok(r) - } - - fn compose_argv( - &self, - build_dir: &BuildDir, - packages: Option<&[&Package]>, - phase: Phase, - options: &Options, - ) -> Result> { - Ok(cargo_argv(build_dir.path(), packages, phase, options)) - } - - fn compose_env(&self) -> Result> { - Ok(self.env.clone()) - } +/// Run cargo build, check, or test. +pub fn run_cargo( + build_dir: &BuildDir, + packages: Option<&[&Package]>, + phase: Phase, + timeout: Duration, + log_file: &mut LogFile, + options: &Options, + console: &Console, +) -> Result { + let _span = debug_span!("run", ?phase).entered(); + let start = Instant::now(); + let argv = cargo_argv(build_dir.path(), packages, phase, options); + let env = vec![ + ("CARGO_ENCODED_RUSTFLAGS".to_owned(), rustflags()), + // The tests might use Insta , and we don't want it to write + // updates to the source tree, and we *certainly* don't want it to write + // updates and then let the test pass. + ("INSTA_UPDATE".to_owned(), "no".to_owned()), + ]; + let process_status = Process::run(&argv, &env, build_dir.path(), timeout, log_file, console)?; + check_interrupted()?; + debug!(?process_status, elapsed = ?start.elapsed()); + Ok(PhaseResult { + phase, + duration: start.elapsed(), + process_status, + argv, + }) } /// Return the name of the cargo binary. -fn cargo_bin() -> String { +pub fn cargo_bin() -> String { // When run as a Cargo subcommand, which is the usual/intended case, // $CARGO tells us the right way to call back into it, so that we get // the matching toolchain etc. @@ -218,51 +121,10 @@ fn rustflags() -> String { rustflags.join("\x1f") } -/// Find all the files that are named in the `path` of targets in a Cargo manifest that should be tested. -/// -/// These are the starting points for discovering source files. -fn direct_package_sources( - workspace_root: &Utf8Path, - package_metadata: &cargo_metadata::Package, -) -> Result> { - let mut found = Vec::new(); - let pkg_dir = package_metadata.manifest_path.parent().unwrap(); - for target in &package_metadata.targets { - if should_mutate_target(target) { - if let Ok(relpath) = target - .src_path - .strip_prefix(workspace_root) - .map(ToOwned::to_owned) - { - debug!( - "found mutation target {} of kind {:?}", - relpath, target.kind - ); - found.push(relpath); - } else { - warn!("{:?} is not in {:?}", target.src_path, pkg_dir); - } - } else { - debug!( - "skipping target {:?} of kinds {:?}", - target.name, target.kind - ); - } - } - found.sort(); - found.dedup(); - Ok(found) -} - -fn should_mutate_target(target: &cargo_metadata::Target) -> bool { - target.kind.iter().any(|k| k.ends_with("lib") || k == "bin") -} - #[cfg(test)] mod test { - use std::ffi::OsStr; + use std::sync::Arc; - use itertools::Itertools; use pretty_assertions::assert_eq; use crate::{Options, Phase}; @@ -360,48 +222,4 @@ mod test { ] ); } - - #[test] - fn error_opening_outside_of_crate() { - CargoTool::new().find_root(Utf8Path::new("/")).unwrap_err(); - } - - #[test] - fn open_subdirectory_of_crate_opens_the_crate() { - let root = CargoTool::new() - .find_root(Utf8Path::new("testdata/tree/factorial/src")) - .expect("open source tree from subdirectory"); - assert!(root.is_dir()); - assert!(root.join("Cargo.toml").is_file()); - assert!(root.join("src/bin/factorial.rs").is_file()); - assert_eq!(root.file_name().unwrap(), OsStr::new("factorial")); - } - - #[test] - fn find_root_from_subdirectory_of_workspace_finds_the_workspace_root() { - let root = CargoTool::new() - .find_root(Utf8Path::new("testdata/tree/workspace/main")) - .expect("Find root from within workspace/main"); - assert_eq!(root.file_name(), Some("workspace"), "Wrong root: {root:?}"); - } - - #[test] - fn find_top_source_files_from_subdirectory_of_workspace() { - let tool = CargoTool::new(); - let root_dir = tool - .find_root(Utf8Path::new("testdata/tree/workspace/main")) - .expect("Find workspace root"); - let top_source_files = tool.top_source_files(&root_dir).expect("Find root files"); - println!("{top_source_files:#?}"); - let paths = top_source_files - .iter() - .map(|sf| sf.tree_relative_path.to_slash_path()) - .collect_vec(); - // The order here might look strange, but they're actually deterministically - // sorted by the package name, not the path name. - assert_eq!( - paths, - ["utils/src/lib.rs", "main/src/main.rs", "main2/src/main.rs"] - ); - } } diff --git a/src/config.rs b/src/config.rs index 620c573e..efcd4d62 100644 --- a/src/config.rs +++ b/src/config.rs @@ -50,8 +50,8 @@ impl Config { /// Read the config from a tree's `.cargo/mutants.toml`, and return a default (empty) /// Config is the file does not exist. - pub fn read_tree_config(source_tree_root: &Utf8Path) -> Result { - let path = source_tree_root.join(".cargo").join("mutants.toml"); + pub fn read_tree_config(workspace_dir: &Utf8Path) -> Result { + let path = workspace_dir.join(".cargo").join("mutants.toml"); if path.exists() { Config::read_file(&path) } else { diff --git a/src/console.rs b/src/console.rs index 508fbc26..ff617332 100644 --- a/src/console.rs +++ b/src/console.rs @@ -272,6 +272,12 @@ impl Console { } } +impl Default for Console { + fn default() -> Self { + Self::new() + } +} + /// Write trace output to the terminal via the console. pub struct TerminalWriter { view: Arc>, diff --git a/src/lab.rs b/src/lab.rs index 3c1aa05f..bb560077 100644 --- a/src/lab.rs +++ b/src/lab.rs @@ -2,31 +2,31 @@ //! Successively apply mutations to the source code and run cargo to check, build, and test them. -use std::cmp::max; +use std::cmp::{max, min}; use std::sync::Mutex; use std::thread; use std::time::{Duration, Instant}; -use anyhow::{anyhow, Context, Result}; +use anyhow::{ensure, Context, Result}; use itertools::Itertools; use tracing::warn; #[allow(unused)] use tracing::{debug, debug_span, error, info, trace}; +use crate::cargo::run_cargo; use crate::console::Console; -use crate::outcome::{LabOutcome, Phase, PhaseResult, ScenarioOutcome}; +use crate::outcome::{LabOutcome, Phase, ScenarioOutcome}; use crate::output::OutputDir; -use crate::process::Process; -use crate::source::Package; +use crate::package::Package; use crate::*; /// Run all possible mutation experiments. /// /// Before testing the mutants, the lab checks that the source tree passes its tests with no /// mutations applied. -pub fn test_unmutated_then_all_mutants( - tool: &dyn Tool, - source_tree: &Utf8Path, +pub fn test_mutants( + mut mutants: Vec, + workspace_dir: &Utf8Path, options: Options, console: &Console, ) -> Result { @@ -34,44 +34,38 @@ pub fn test_unmutated_then_all_mutants( let output_in_dir: &Utf8Path = options .output_in_dir .as_ref() - .map_or(source_tree, |p| p.as_path()); + .map_or(workspace_dir, |p| p.as_path()); let output_dir = OutputDir::new(output_in_dir)?; console.set_debug_log(output_dir.open_debug_log()?); - let mut mutants = walk_tree(tool, source_tree, &options, console)?.mutants; if options.shuffle { fastrand::shuffle(&mut mutants); } output_dir.write_mutants_list(&mutants)?; console.discovered_mutants(&mutants); - if mutants.is_empty() { - return Err(anyhow!("No mutants found")); - } + ensure!(!mutants.is_empty(), "No mutants found"); let all_packages = mutants.iter().map(|m| m.package()).unique().collect_vec(); + debug!(?all_packages); let output_mutex = Mutex::new(output_dir); - let mut build_dirs = vec![BuildDir::new(source_tree, &options, console)?]; + let mut build_dirs = vec![BuildDir::new(workspace_dir, &options, console)?]; let baseline_outcome = { let _span = debug_span!("baseline").entered(); test_scenario( - tool, &mut build_dirs[0], &output_mutex, - &options, &Scenario::Baseline, &all_packages, options.test_timeout.unwrap_or(Duration::MAX), + &options, console, )? }; if !baseline_outcome.success() { error!( - "{} {} failed in an unmutated tree, so no mutants were tested", - tool.name(), + "cargo {} failed in an unmutated tree, so no mutants were tested", baseline_outcome.last_phase(), ); - // TODO: Maybe should be Err, but it would need to be an error that can map to the right - // exit code. return Ok(output_mutex .into_inner() .expect("lock output_dir") @@ -98,7 +92,7 @@ pub fn test_unmutated_then_all_mutants( Duration::MAX }; - let jobs = std::cmp::max(1, std::cmp::min(options.jobs.unwrap_or(1), mutants.len())); + let jobs = max(1, min(options.jobs.unwrap_or(1), mutants.len())); console.build_dirs_start(jobs - 1); for i in 1..jobs { debug!("copy build dir {i}"); @@ -127,13 +121,12 @@ pub fn test_unmutated_then_all_mutants( let package = mutant.package().clone(); // We don't care about the outcome; it's been collected into the output_dir. let _outcome = test_scenario( - tool, &mut build_dir, &output_mutex, - &options, &Scenario::Mutant(mutant), &[&package], mutated_test_timeout, + &options, console, ) .expect("scenario test"); @@ -168,21 +161,13 @@ pub fn test_unmutated_then_all_mutants( /// /// The [BuildDir] is passed as mutable because it's for the exclusive use of this function for the /// duration of the test. -#[allow( - unknown_lints, - clippy::needless_pass_by_ref_mut, - clippy::too_many_arguments -)] -// Yes, it's a lot of arguments, but it does use them all and I don't think creating objects -// just to group them would help... fn test_scenario( - tool: &dyn Tool, build_dir: &mut BuildDir, output_mutex: &Mutex, - options: &Options, scenario: &Scenario, - packages: &[&Package], + test_packages: &[&Package], test_timeout: Duration, + options: &Options, console: &Console, ) -> Result { let mut log_file = output_mutex @@ -203,34 +188,24 @@ fn test_scenario( &[Phase::Build, Phase::Test] }; for &phase in phases { - let _span = debug_span!("run", ?phase).entered(); - let start = Instant::now(); console.scenario_phase_started(scenario, phase); let timeout = match phase { Phase::Test => test_timeout, _ => Duration::MAX, }; - let argv = tool.compose_argv(build_dir, Some(packages), phase, options)?; - let env = tool.compose_env()?; - let process_status = Process::run( - &argv, - &env, - build_dir.path(), + let phase_result = run_cargo( + build_dir, + Some(test_packages), + phase, timeout, &mut log_file, + options, console, )?; - check_interrupted()?; - debug!(?process_status, elapsed = ?start.elapsed()); - let phase_result = PhaseResult { - phase, - duration: start.elapsed(), - process_status, - argv, - }; + let success = phase_result.is_success(); outcome.add_phase_result(phase_result); console.scenario_phase_finished(scenario, phase); - if (phase == Phase::Check && options.check_only) || !process_status.success() { + if (phase == Phase::Check && options.check_only) || !success { break; } } diff --git a/src/list.rs b/src/list.rs index 9eae357c..1b9edcfd 100644 --- a/src/list.rs +++ b/src/list.rs @@ -5,13 +5,12 @@ use std::fmt; use std::io; -use camino::Utf8Path; use serde_json::{json, Value}; use crate::console::style_mutant; -use crate::console::Console; use crate::path::Utf8PathSlashes; -use crate::{walk_tree, Options, Result, Tool}; +use crate::visit::Discovered; +use crate::{Options, Result}; /// Convert `fmt::Write` to `io::Write`. pub(crate) struct FmtToIoWrite(W); @@ -30,13 +29,9 @@ impl fmt::Write for FmtToIoWrite { pub(crate) fn list_mutants( mut out: W, - tool: &dyn Tool, - source_tree_root: &Utf8Path, + discovered: Discovered, options: &Options, - console: &Console, ) -> Result<()> { - let discovered = walk_tree(tool, source_tree_root, options, console)?; - console.clear(); if options.emit_json { let mut list: Vec = Vec::new(); for mutant in discovered.mutants { @@ -66,15 +61,13 @@ pub(crate) fn list_mutants( pub(crate) fn list_files( mut out: W, - tool: &dyn Tool, - source: &Utf8Path, + discovered: Discovered, options: &Options, - console: &Console, ) -> Result<()> { - let files = walk_tree(tool, source, options, console)?.files; if options.emit_json { let json_list = Value::Array( - files + discovered + .files .iter() .map(|source_file| { json!({ @@ -86,7 +79,7 @@ pub(crate) fn list_files( ); writeln!(out, "{}", serde_json::to_string_pretty(&json_list)?)?; } else { - for file in files { + for file in discovered.files { writeln!(out, "{}", file.tree_relative_path.to_slash_path())?; } } diff --git a/src/main.rs b/src/main.rs index 35720f15..dafe78b8 100644 --- a/src/main.rs +++ b/src/main.rs @@ -17,14 +17,15 @@ mod mutate; mod options; mod outcome; mod output; +mod package; mod path; mod pretty; mod process; mod scenario; mod source; mod textedit; -mod tool; mod visit; +pub mod workspace; use std::env; use std::io; @@ -41,9 +42,9 @@ use tracing::debug; // Imports of public names from this crate. use crate::build_dir::BuildDir; -use crate::cargo::CargoTool; use crate::console::Console; use crate::interrupt::check_interrupted; +use crate::lab::test_mutants; use crate::list::{list_files, list_mutants, FmtToIoWrite}; use crate::log_file::{last_line, LogFile}; use crate::manifest::fix_manifest; @@ -52,9 +53,7 @@ use crate::options::Options; use crate::outcome::{Phase, ScenarioOutcome}; use crate::path::Utf8PathSlashes; use crate::scenario::Scenario; -use crate::source::SourceFile; -use crate::tool::Tool; -use crate::visit::walk_tree; +use crate::workspace::{PackageFilter, Workspace}; const VERSION: &str = env!("CARGO_PKG_VERSION"); const NAME: &str = env!("CARGO_PKG_NAME"); @@ -164,6 +163,10 @@ struct Args { #[arg(long, short = 'o')] output: Option, + /// only test mutants from these packages. + #[arg(id = "package", long, short = 'p')] + mutate_packages: Vec, + /// run mutants in random order. #[arg(long)] shuffle: bool, @@ -188,6 +191,10 @@ struct Args { #[arg(long, action = clap::ArgAction::SetTrue)] version: bool, + /// test every package in the workspace. + #[arg(long)] + workspace: bool, + /// additional args for all cargo invocations. #[arg(long, short = 'C', allow_hyphen_values = true)] cargo_arg: Vec, @@ -207,9 +214,6 @@ fn main() -> Result<()> { exit(exit_code::USAGE); } }; - let console = Console::new(); - console.setup_global_trace(args.level)?; - interrupt::install_handler(); if args.version { println!("{NAME} {VERSION}"); @@ -219,41 +223,38 @@ fn main() -> Result<()> { return Ok(()); } - let source_path: &Utf8Path = if let Some(p) = &args.dir { - p + let console = Console::new(); + console.setup_global_trace(args.level)?; + interrupt::install_handler(); + + let start_dir: &Utf8Path = args.dir.as_deref().unwrap_or(Utf8Path::new(".")); + let workspace = Workspace::open(start_dir)?; + // let discovered_workspace = discover_packages(start_dir, false, &args.mutate_packages)?; + // let workspace_dir = &discovered_workspace.workspace_dir; + let config = if args.no_config { + config::Config::default() } else { - Utf8Path::new(".") + config::Config::read_tree_config(&workspace.dir)? }; - let tool = CargoTool::new(); - let source_tree_root = tool.find_root(source_path)?; - let config; - if args.no_config { - config = config::Config::default(); - } else { - config = config::Config::read_tree_config(&source_tree_root)?; - debug!(?config); - } + debug!(?config); let options = Options::new(&args, &config)?; debug!(?options); + let package_filter = if !args.mutate_packages.is_empty() { + PackageFilter::explicit(&args.mutate_packages) + } else if args.workspace { + PackageFilter::All + } else { + PackageFilter::Auto(start_dir.to_owned()) + }; + let discovered = workspace.discover(&package_filter, &options, &console)?; if args.list_files { - list_files( - FmtToIoWrite::new(io::stdout()), - &tool, - &source_tree_root, - &options, - &console, - )?; + console.clear(); + list_files(FmtToIoWrite::new(io::stdout()), discovered, &options)?; } else if args.list { - list_mutants( - FmtToIoWrite::new(io::stdout()), - &tool, - &source_tree_root, - &options, - &console, - )?; + console.clear(); + list_mutants(FmtToIoWrite::new(io::stdout()), discovered, &options)?; } else { - let lab_outcome = - lab::test_unmutated_then_all_mutants(&tool, &source_tree_root, options, &console)?; + let lab_outcome = test_mutants(discovered.mutants, &workspace.dir, options, &console)?; exit(lab_outcome.exit_code()); } Ok(()) diff --git a/src/manifest.rs b/src/manifest.rs index 5061abb0..15b8db8c 100644 --- a/src/manifest.rs +++ b/src/manifest.rs @@ -161,7 +161,7 @@ mod test { use pretty_assertions::assert_eq; use toml::Table; - use super::fix_manifest_toml; + use super::{fix_cargo_config_toml, fix_manifest_toml}; #[test] fn fix_path_absolute_unchanged() { @@ -305,7 +305,7 @@ mod test { "/src/other", ]"# }; let source_dir = Utf8Path::new("/Users/jane/src/foo"); - let fixed_toml = super::fix_cargo_config_toml(cargo_config_toml, source_dir) + let fixed_toml = fix_cargo_config_toml(cargo_config_toml, source_dir) .unwrap() .expect("toml was modified"); println!("fixed toml:\n{fixed_toml}"); diff --git a/src/mutate.rs b/src/mutate.rs index eb427791..9c56f457 100644 --- a/src/mutate.rs +++ b/src/mutate.rs @@ -14,7 +14,7 @@ use serde::Serialize; use similar::TextDiff; use crate::build_dir::BuildDir; -use crate::source::Package; +use crate::package::Package; use crate::source::SourceFile; use crate::textedit::{replace_region, Span}; @@ -121,15 +121,18 @@ impl Mutant { .to_string() } - pub fn apply(&self, build_dir: &BuildDir) -> Result<()> { + /// Apply this mutant to the relevant file within a BuildDir. + pub fn apply(&self, build_dir: &mut BuildDir) -> Result<()> { self.write_in_dir(build_dir, &self.mutated_code()) } - pub fn unapply(&self, build_dir: &BuildDir) -> Result<()> { + pub fn unapply(&self, build_dir: &mut BuildDir) -> Result<()> { self.write_in_dir(build_dir, self.original_code()) } - fn write_in_dir(&self, build_dir: &BuildDir, code: &str) -> Result<()> { + #[allow(unknown_lints, clippy::needless_pass_by_ref_mut)] + // The Rust object is not mutated, but the BuildDir on disk should be exclusively owned for this to be safe. + fn write_in_dir(&self, build_dir: &mut BuildDir, code: &str) -> Result<()> { let path = build_dir.path().join(&self.source_file.tree_relative_path); // for safety, don't follow symlinks ensure!(path.is_file(), "{path:?} is not a file"); @@ -208,12 +211,11 @@ mod test { #[test] fn discover_factorial_mutants() { let tree_path = Utf8Path::new("testdata/tree/factorial"); - let tool = CargoTool::new(); - let source_tree = tool.find_root(tree_path).unwrap(); + let workspace = Workspace::open(tree_path).unwrap(); let options = Options::default(); - let mutants = walk_tree(&tool, &source_tree, &options, &Console::new()) - .unwrap() - .mutants; + let mutants = workspace + .mutants(&PackageFilter::All, &options, &Console::new()) + .unwrap(); assert_eq!(mutants.len(), 3); assert_eq!( format!("{:?}", mutants[0]), @@ -262,12 +264,10 @@ mod test { #[test] fn filter_by_attributes() { - let tree_path = Utf8Path::new("testdata/tree/hang_avoided_by_attr"); - let tool = CargoTool::new(); - let source_tree = tool.find_root(tree_path).unwrap(); - let mutants = walk_tree(&tool, &source_tree, &Options::default(), &Console::new()) + let mutants = Workspace::open(Utf8Path::new("testdata/tree/hang_avoided_by_attr")) .unwrap() - .mutants; + .mutants(&PackageFilter::All, &Options::default(), &Console::new()) + .unwrap(); let descriptions = mutants.iter().map(Mutant::describe_change).collect_vec(); insta::assert_snapshot!( descriptions.join("\n"), @@ -276,13 +276,13 @@ mod test { } #[test] - fn mutate_factorial() { + fn mutate_factorial() -> Result<()> { let tree_path = Utf8Path::new("testdata/tree/factorial"); - let tool = CargoTool::new(); - let source_tree = tool.find_root(tree_path).unwrap(); - let mutants = walk_tree(&tool, &source_tree, &Options::default(), &Console::new()) - .unwrap() - .mutants; + let mutants = Workspace::open(tree_path)?.mutants( + &PackageFilter::All, + &Options::default(), + &Console::new(), + )?; assert_eq!(mutants.len(), 3); let mut mutated_code = mutants[0].mutated_code(); @@ -336,5 +336,6 @@ mod test { "# } ); + Ok(()) } } diff --git a/src/outcome.rs b/src/outcome.rs index daeb5751..3505a69b 100644 --- a/src/outcome.rs +++ b/src/outcome.rs @@ -273,6 +273,12 @@ pub struct PhaseResult { pub argv: Vec, } +impl PhaseResult { + pub fn is_success(&self) -> bool { + self.process_status.success() + } +} + impl Serialize for PhaseResult { fn serialize(&self, serializer: S) -> Result where diff --git a/src/output.rs b/src/output.rs index 60716529..54e44183 100644 --- a/src/output.rs +++ b/src/output.rs @@ -279,13 +279,14 @@ mod test { #[test] fn create_output_dir() { let tmp = minimal_source_tree(); - let tmp_path = tmp.path().try_into().unwrap(); - let root = CargoTool::new().find_root(tmp_path).unwrap(); - let output_dir = OutputDir::new(&root).unwrap(); + let tmp_path: &Utf8Path = tmp.path().try_into().unwrap(); + let workspace = Workspace::open(tmp_path).unwrap(); + let output_dir = OutputDir::new(&workspace.dir).unwrap(); assert_eq!( list_recursive(tmp.path()), &[ "", + "Cargo.lock", "Cargo.toml", "mutants.out", "mutants.out/caught.txt", @@ -298,8 +299,8 @@ mod test { "src/lib.rs", ] ); - assert_eq!(output_dir.path(), root.join("mutants.out")); - assert_eq!(output_dir.log_dir, root.join("mutants.out/log")); + assert_eq!(output_dir.path(), workspace.dir.join("mutants.out")); + assert_eq!(output_dir.log_dir, workspace.dir.join("mutants.out/log")); assert!(output_dir.path().join("lock.json").is_file()); } diff --git a/src/package.rs b/src/package.rs new file mode 100644 index 00000000..3bb30947 --- /dev/null +++ b/src/package.rs @@ -0,0 +1,15 @@ +// Copyright 2023 Martin Pool + +//! Discover and represent cargo packages within a workspace. + +use camino::Utf8PathBuf; + +/// A package built and tested as a unit. +#[derive(Debug, Eq, PartialEq, Hash, Clone)] +pub struct Package { + /// The short name of the package, like "mutants". + pub name: String, + + /// For Cargo, the path of the `Cargo.toml` manifest file, relative to the top of the tree. + pub relative_manifest_path: Utf8PathBuf, +} diff --git a/src/snapshots/cargo_mutants__visit__test__expected_mutants_for_own_source_tree.snap b/src/snapshots/cargo_mutants__visit__test__expected_mutants_for_own_source_tree.snap index 9689f8d3..73524271 100644 --- a/src/snapshots/cargo_mutants__visit__test__expected_mutants_for_own_source_tree.snap +++ b/src/snapshots/cargo_mutants__visit__test__expected_mutants_for_own_source_tree.snap @@ -11,23 +11,8 @@ src/build_dir.rs: replace ::fmt -> fmt::Result with Ok( src/build_dir.rs: replace ::fmt -> fmt::Result with Err(::anyhow::anyhow!("mutated!")) src/build_dir.rs: replace copy_tree -> Result with Ok(Default::default()) src/build_dir.rs: replace copy_tree -> Result with Err(::anyhow::anyhow!("mutated!")) -src/cargo.rs: replace ::name -> &str with "" -src/cargo.rs: replace ::name -> &str with "xyzzy" -src/cargo.rs: replace ::find_root -> Result with Ok(Default::default()) -src/cargo.rs: replace ::find_root -> Result with Err(::anyhow::anyhow!("mutated!")) -src/cargo.rs: replace ::top_source_files -> Result>> with Ok(vec![]) -src/cargo.rs: replace ::top_source_files -> Result>> with Ok(vec![Arc::new(Default::default())]) -src/cargo.rs: replace ::top_source_files -> Result>> with Err(::anyhow::anyhow!("mutated!")) -src/cargo.rs: replace ::compose_argv -> Result> with Ok(vec![]) -src/cargo.rs: replace ::compose_argv -> Result> with Ok(vec![String::new()]) -src/cargo.rs: replace ::compose_argv -> Result> with Ok(vec!["xyzzy".into()]) -src/cargo.rs: replace ::compose_argv -> Result> with Err(::anyhow::anyhow!("mutated!")) -src/cargo.rs: replace ::compose_env -> Result> with Ok(vec![]) -src/cargo.rs: replace ::compose_env -> Result> with Ok(vec![(String::new(), String::new())]) -src/cargo.rs: replace ::compose_env -> Result> with Ok(vec![(String::new(), "xyzzy".into())]) -src/cargo.rs: replace ::compose_env -> Result> with Ok(vec![("xyzzy".into(), String::new())]) -src/cargo.rs: replace ::compose_env -> Result> with Ok(vec![("xyzzy".into(), "xyzzy".into())]) -src/cargo.rs: replace ::compose_env -> Result> with Err(::anyhow::anyhow!("mutated!")) +src/cargo.rs: replace run_cargo -> Result with Ok(Default::default()) +src/cargo.rs: replace run_cargo -> Result with Err(::anyhow::anyhow!("mutated!")) src/cargo.rs: replace cargo_bin -> String with String::new() src/cargo.rs: replace cargo_bin -> String with "xyzzy".into() src/cargo.rs: replace cargo_argv -> Vec with vec![] @@ -35,11 +20,6 @@ src/cargo.rs: replace cargo_argv -> Vec with vec![String::new()] src/cargo.rs: replace cargo_argv -> Vec with vec!["xyzzy".into()] src/cargo.rs: replace rustflags -> String with String::new() src/cargo.rs: replace rustflags -> String with "xyzzy".into() -src/cargo.rs: replace direct_package_sources -> Result> with Ok(vec![]) -src/cargo.rs: replace direct_package_sources -> Result> with Ok(vec![Default::default()]) -src/cargo.rs: replace direct_package_sources -> Result> with Err(::anyhow::anyhow!("mutated!")) -src/cargo.rs: replace should_mutate_target -> bool with true -src/cargo.rs: replace should_mutate_target -> bool with false src/config.rs: replace Config::read_file -> Result with Ok(Default::default()) src/config.rs: replace Config::read_file -> Result with Err(::anyhow::anyhow!("mutated!")) src/config.rs: replace Config::read_tree_config -> Result with Ok(Default::default()) @@ -153,8 +133,8 @@ src/fnvalue.rs: replace path_is_nonzero_unsigned -> bool with false src/fnvalue.rs: replace match_first_type_arg -> Option<&'p Type> with None src/fnvalue.rs: replace match_first_type_arg -> Option<&'p Type> with Some(&Default::default()) src/interrupt.rs: replace install_handler with () -src/lab.rs: replace test_unmutated_then_all_mutants -> Result with Ok(Default::default()) -src/lab.rs: replace test_unmutated_then_all_mutants -> Result with Err(::anyhow::anyhow!("mutated!")) +src/lab.rs: replace test_mutants -> Result with Ok(Default::default()) +src/lab.rs: replace test_mutants -> Result with Err(::anyhow::anyhow!("mutated!")) src/lab.rs: replace test_scenario -> Result with Ok(Default::default()) src/lab.rs: replace test_scenario -> Result with Err(::anyhow::anyhow!("mutated!")) src/list.rs: replace >::write_str -> Result<(), fmt::Error> with Ok(()) @@ -259,6 +239,8 @@ src/outcome.rs: replace ScenarioOutcome::mutant_caught -> bool with false src/outcome.rs: replace ScenarioOutcome::mutant_missed -> bool with true src/outcome.rs: replace ScenarioOutcome::mutant_missed -> bool with false src/outcome.rs: replace ScenarioOutcome::summary -> SummaryOutcome with Default::default() +src/outcome.rs: replace PhaseResult::is_success -> bool with true +src/outcome.rs: replace PhaseResult::is_success -> bool with false src/outcome.rs: replace ::serialize -> Result with Ok(Default::default()) src/outcome.rs: replace ::serialize -> Result with Err(::anyhow::anyhow!("mutated!")) src/output.rs: replace LockFile::acquire_lock -> Result with Ok(Default::default()) @@ -343,4 +325,30 @@ src/visit.rs: replace path_is -> bool with true src/visit.rs: replace path_is -> bool with false src/visit.rs: replace attr_is_mutants_skip -> bool with true src/visit.rs: replace attr_is_mutants_skip -> bool with false +src/workspace.rs: replace PackageFilter::explicit -> PackageFilter with Default::default() +src/workspace.rs: replace PackageFilter::resolve_auto -> Result with Ok(Default::default()) +src/workspace.rs: replace PackageFilter::resolve_auto -> Result with Err(::anyhow::anyhow!("mutated!")) +src/workspace.rs: replace Workspace::open -> Result with Ok(Default::default()) +src/workspace.rs: replace Workspace::open -> Result with Err(::anyhow::anyhow!("mutated!")) +src/workspace.rs: replace Workspace::packages -> Result>> with Ok(vec![]) +src/workspace.rs: replace Workspace::packages -> Result>> with Ok(vec![Arc::new(Default::default())]) +src/workspace.rs: replace Workspace::packages -> Result>> with Err(::anyhow::anyhow!("mutated!")) +src/workspace.rs: replace Workspace::package_tops -> Result> with Ok(vec![]) +src/workspace.rs: replace Workspace::package_tops -> Result> with Ok(vec![Default::default()]) +src/workspace.rs: replace Workspace::package_tops -> Result> with Err(::anyhow::anyhow!("mutated!")) +src/workspace.rs: replace Workspace::top_sources -> Result>> with Ok(vec![]) +src/workspace.rs: replace Workspace::top_sources -> Result>> with Ok(vec![Arc::new(Default::default())]) +src/workspace.rs: replace Workspace::top_sources -> Result>> with Err(::anyhow::anyhow!("mutated!")) +src/workspace.rs: replace Workspace::discover -> Result with Ok(Default::default()) +src/workspace.rs: replace Workspace::discover -> Result with Err(::anyhow::anyhow!("mutated!")) +src/workspace.rs: replace Workspace::mutants -> Result> with Ok(vec![]) +src/workspace.rs: replace Workspace::mutants -> Result> with Ok(vec![Default::default()]) +src/workspace.rs: replace Workspace::mutants -> Result> with Err(::anyhow::anyhow!("mutated!")) +src/workspace.rs: replace direct_package_sources -> Result> with Ok(vec![]) +src/workspace.rs: replace direct_package_sources -> Result> with Ok(vec![Default::default()]) +src/workspace.rs: replace direct_package_sources -> Result> with Err(::anyhow::anyhow!("mutated!")) +src/workspace.rs: replace should_mutate_target -> bool with true +src/workspace.rs: replace should_mutate_target -> bool with false +src/workspace.rs: replace locate_project -> Result with Ok(Default::default()) +src/workspace.rs: replace locate_project -> Result with Err(::anyhow::anyhow!("mutated!")) diff --git a/src/source.rs b/src/source.rs index 44eaa173..bf5ee6e5 100644 --- a/src/source.rs +++ b/src/source.rs @@ -9,6 +9,7 @@ use camino::{Utf8Path, Utf8PathBuf}; #[allow(unused_imports)] use tracing::{debug, info, warn}; +use crate::package::Package; use crate::path::Utf8PathSlashes; /// A Rust source file within a source tree. @@ -23,11 +24,11 @@ pub struct SourceFile { /// Package within the workspace. pub package: Arc, - /// Path relative to the root of the tree. + /// Path of this source file relative to workspace. pub tree_relative_path: Utf8PathBuf, /// Full copy of the source. - pub code: Arc, + pub code: String, } impl SourceFile { @@ -45,7 +46,7 @@ impl SourceFile { .replace("\r\n", "\n"); Ok(SourceFile { tree_relative_path, - code: Arc::new(code), + code, package: Arc::clone(package), }) } @@ -56,15 +57,6 @@ impl SourceFile { } } -/// A package built and tested as a unit. -#[derive(Debug, Eq, PartialEq, Hash, Clone)] -pub struct Package { - /// The short name of the package, like "mutants". - pub name: String, - /// For Cargo, the path of the `Cargo.toml` manifest file, relative to the top of the tree. - pub relative_manifest_path: Utf8PathBuf, -} - #[cfg(test)] mod test { use std::fs::File; @@ -93,6 +85,6 @@ mod test { }), ) .unwrap(); - assert_eq!(*source_file.code, "fn main() {\n 640 << 10;\n}\n"); + assert_eq!(source_file.code, "fn main() {\n 640 << 10;\n}\n"); } } diff --git a/src/tool.rs b/src/tool.rs deleted file mode 100644 index 96cdbcfd..00000000 --- a/src/tool.rs +++ /dev/null @@ -1,57 +0,0 @@ -// Copyright 2023 Martin Pool. - -//! A generic interface for running a tool such as Cargo that can determine the tree -//! shape, build it, and run tests. -//! -//! At present only Cargo is supported, but this interface aims to leave a place to -//! support for example Bazel in future. - -use std::fmt::Debug; -use std::marker::{Send, Sync}; -use std::sync::Arc; - -use camino::{Utf8Path, Utf8PathBuf}; -#[allow(unused_imports)] -use tracing::{debug, debug_span, trace}; - -use crate::options::Options; -use crate::outcome::Phase; -use crate::source::Package; -use crate::SourceFile; -use crate::{build_dir, Result}; - -pub trait Tool: Debug + Send + Sync { - /// A short name for this tool, like "cargo". - fn name(&self) -> &str; - - /// Find the root of the source tree enclosing a given path. - /// - /// The root is the enclosing directory that needs to be copied to make a self-contained - /// scratch directory, and from where source discovery begins. - /// - /// This may include more directories than will actually be tested, sufficient to allow - /// the build to work. For Cargo, we copy the whole workspace. - fn find_root(&self, path: &Utf8Path) -> Result; - - /// Find the top-level files for each package within a tree. - /// - /// The path is the root returned by [find_root]. - /// - /// For Cargo, this is files like `src/bin/*.rs`, `src/lib.rs` identified by targets - /// in the manifest for each package. - /// - /// From each of these top files, we can discover more source by following `mod` - /// statements. - fn top_source_files(&self, path: &Utf8Path) -> Result>>; - - /// Compose argv to run one phase in this tool. - fn compose_argv( - &self, - build_dir: &build_dir::BuildDir, - packages: Option<&[&Package]>, - phase: Phase, - options: &Options, - ) -> Result>; - - fn compose_env(&self) -> Result>; -} diff --git a/src/visit.rs b/src/visit.rs index 1046dfe4..35a63cc7 100644 --- a/src/visit.rs +++ b/src/visit.rs @@ -34,21 +34,23 @@ pub struct Discovered { /// Discover all mutants and all source files. /// /// The list of source files includes even those with no mutants. +/// pub fn walk_tree( - tool: &dyn Tool, - root: &Utf8Path, + workspace_dir: &Utf8Path, + top_source_files: &[Arc], options: &Options, console: &Console, ) -> Result { + // TODO: Lift up parsing the error expressions... let error_exprs = options .error_values .iter() .map(|e| syn::parse_str(e).with_context(|| format!("Failed to parse error value {e:?}"))) .collect::>>()?; console.walk_tree_start(); + let mut file_queue: VecDeque> = top_source_files.iter().cloned().collect(); let mut mutants = Vec::new(); let mut files: Vec> = Vec::new(); - let mut file_queue: VecDeque> = tool.top_source_files(root)?.into(); while let Some(source_file) = file_queue.pop_front() { console.walk_tree_update(files.len(), mutants.len()); check_interrupted()?; @@ -58,9 +60,9 @@ pub fn walk_tree( // collect any mutants from them, and they don't count as "seen" for // `--list-files`. for mod_name in &external_mods { - if let Some(mod_path) = find_mod_source(root, &source_file, mod_name)? { + if let Some(mod_path) = find_mod_source(workspace_dir, &source_file, mod_name)? { file_queue.push_back(Arc::new(SourceFile::new( - root, + workspace_dir, mod_path, &source_file.package, )?)) @@ -393,8 +395,6 @@ fn attr_is_mutants_skip(attr: &Attribute) -> bool { mod test { use regex::Regex; - use crate::cargo::CargoTool; - use super::*; /// As a generic protection against regressions in discovery, the the mutants @@ -411,16 +411,18 @@ mod test { ..Default::default() }; let mut list_output = String::new(); - crate::list_mutants( - &mut list_output, - &CargoTool::new(), - &Utf8Path::new(".") + let console = Console::new(); + let workspace = Workspace::open( + Utf8Path::new(".") .canonicalize_utf8() .expect("Canonicalize source path"), - &options, - &Console::new(), ) - .expect("Discover mutants in own source tree"); + .unwrap(); + let discovered = workspace + .discover(&PackageFilter::All, &options, &console) + .expect("Discover mutants"); + crate::list_mutants(&mut list_output, discovered, &options) + .expect("Discover mutants in own source tree"); // Strip line numbers so this is not too brittle. let line_re = Regex::new(r"(?m)^([^:]+:)\d+:( .*)$").unwrap(); diff --git a/src/workspace.rs b/src/workspace.rs new file mode 100644 index 00000000..050f65ff --- /dev/null +++ b/src/workspace.rs @@ -0,0 +1,446 @@ +// Copyright 2023 Martin Pool + +use std::fmt; +use std::panic::catch_unwind; +use std::sync::Arc; + +use anyhow::{anyhow, ensure, Context}; +use camino::{Utf8Path, Utf8PathBuf}; +use itertools::Itertools; +use serde_json::Value; +use tracing::{debug, debug_span, warn}; + +use crate::cargo::cargo_bin; +use crate::console::Console; +use crate::interrupt::check_interrupted; +use crate::mutate::Mutant; +use crate::options::Options; +use crate::package::Package; +use crate::process::get_command_output; +use crate::source::SourceFile; +use crate::visit::{walk_tree, Discovered}; +use crate::Result; + +pub struct Workspace { + pub dir: Utf8PathBuf, + metadata: cargo_metadata::Metadata, +} + +impl fmt::Debug for Workspace { + #[mutants::skip] + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Workspace") + .field("dir", &self.dir) + // .field("metadata", &self.metadata) + .finish() + } +} + +/// Which packages to mutate in a workspace? +#[derive(Debug, Clone)] +pub enum PackageFilter { + /// Include every package in the workspace. + All, + /// Packages with given names, from `--package`. + Explicit(Vec), + /// Automatic behavior when invoked from a subdirectory, as per + /// . + /// + /// If the directory is within a package directory, select that package. + /// + /// Otherwise, this is a "virtual workspace" directory, containing members but no + /// primary package. In this case, if there is a `default-members` field in the workspace, + /// use that list. Otherwise, apply to all members of the workspace. + Auto(Utf8PathBuf), +} + +impl PackageFilter { + pub fn explicit>(names: I) -> PackageFilter { + PackageFilter::Explicit(names.into_iter().map(|s| s.to_string()).collect_vec()) + } + + /// Translate an auto package filter to either All or Explicit. + pub fn resolve_auto(&self, metadata: &cargo_metadata::Metadata) -> Result { + if let PackageFilter::Auto(dir) = &self { + let package_dir = locate_project(dir, false)?; + let workspace_dir = &metadata.workspace_root; + // It's not required that the members be inside the workspace directory: see + // + for package in metadata.workspace_packages() { + if package.manifest_path.parent().expect("remove Cargo.toml") == package_dir { + debug!("resolved auto package filter to {:?}", package.name); + return Ok(PackageFilter::explicit([&package.name])); + } + } + // Presumably our manifest is the workspace root manifest and there is no + // top-level package? + ensure!( + &package_dir == workspace_dir, + "package {package_dir:?} doesn't match any child and doesn't match the workspace root {workspace_dir:?}?", + ); + // `workspace_default_packages` will panic when calling Cargo older than 1.71; + // in that case we'll just fall back to everything, for lack of a better option. + match catch_unwind(|| metadata.workspace_default_packages()) { + Ok(dm) if dm.is_empty() => Ok(PackageFilter::All), + Ok(dm) => Ok(PackageFilter::explicit( + dm.into_iter().map(|pmeta| &pmeta.name), + )), + Err(err) => { + warn!( + cargo_metadata_error = + err.downcast::().expect("panic message is a string"), + "workspace_default_packages is not supported; testing all packages", + ); + Ok(PackageFilter::All) + } + } + } else { + Ok(self.clone()) + } + } +} + +/// A package and the top source files within it. +struct PackageTop { + package: Arc, + top_sources: Vec, +} + +impl Workspace { + pub fn open>(start_dir: P) -> Result { + let dir = locate_project(start_dir.as_ref(), true)?; + let cargo_toml_path = dir.join("Cargo.toml"); + debug!(?cargo_toml_path, ?dir, "Find root files"); + check_interrupted()?; + let metadata = cargo_metadata::MetadataCommand::new() + .manifest_path(&cargo_toml_path) + .exec() + .context("run cargo metadata")?; + Ok(Workspace { dir, metadata }) + } + + /// Find packages to mutate, subject to some filtering. + pub fn packages(&self, package_filter: &PackageFilter) -> Result>> { + Ok(self + .package_tops(package_filter)? + .into_iter() + .map(|pt| pt.package) + .collect()) + } + + /// Find all the packages and their top source files. + fn package_tops(&self, package_filter: &PackageFilter) -> Result> { + let mut tops = Vec::new(); + let package_filter = package_filter.resolve_auto(&self.metadata)?; + for package_metadata in self + .metadata + .workspace_packages() + .into_iter() + .sorted_by_key(|p| &p.name) + { + check_interrupted()?; + let name = &package_metadata.name; + let _span = debug_span!("package", %name).entered(); + if let PackageFilter::Explicit(ref include_names) = package_filter { + if !include_names.contains(name) { + continue; + } + } + let manifest_path = &package_metadata.manifest_path; + debug!(%manifest_path, "walk package"); + let relative_manifest_path = manifest_path + .strip_prefix(&self.dir) + .map_err(|_| { + anyhow!( + "manifest path {manifest_path:?} for package {name:?} is not \ + within the detected source root path {dir:?}", + dir = self.dir + ) + })? + .to_owned(); + let package = Arc::new(Package { + name: package_metadata.name.clone(), + relative_manifest_path, + }); + tops.push(PackageTop { + package, + top_sources: direct_package_sources(&self.dir, package_metadata)?, + }); + } + if let PackageFilter::Explicit(ref names) = package_filter { + for wanted in names { + if !tops.iter().any(|found| found.package.name == *wanted) { + warn!("package {wanted:?} not found in source tree"); + } + } + } + Ok(tops) + } + + /// Find all the top source files for selected packages. + fn top_sources(&self, package_filter: &PackageFilter) -> Result>> { + let mut sources = Vec::new(); + for PackageTop { + package, + top_sources, + } in self.package_tops(package_filter)? + { + for source_path in top_sources { + sources.push(Arc::new(SourceFile::new( + &self.dir, + source_path.to_owned(), + &package, + )?)); + } + } + Ok(sources) + } + + /// Make all the mutants from the filtered packages in this workspace. + pub fn discover( + &self, + package_filter: &PackageFilter, + options: &Options, + console: &Console, + ) -> Result { + walk_tree( + &self.dir, + &self.top_sources(package_filter)?, + options, + console, + ) + } + + /// Return all mutants generated from this workspace. + pub fn mutants( + &self, + package_filter: &PackageFilter, + options: &Options, + console: &Console, + ) -> Result> { + Ok(self.discover(package_filter, options, console)?.mutants) + } +} + +/// Find all the files that are named in the `path` of targets in a Cargo manifest that should be tested. +/// +/// These are the starting points for discovering source files. +fn direct_package_sources( + workspace_root: &Utf8Path, + package_metadata: &cargo_metadata::Package, +) -> Result> { + let mut found = Vec::new(); + let pkg_dir = package_metadata.manifest_path.parent().unwrap(); + for target in &package_metadata.targets { + if should_mutate_target(target) { + if let Ok(relpath) = target + .src_path + .strip_prefix(workspace_root) + .map(ToOwned::to_owned) + { + debug!( + "found mutation target {} of kind {:?}", + relpath, target.kind + ); + found.push(relpath); + } else { + warn!("{:?} is not in {:?}", target.src_path, pkg_dir); + } + } else { + debug!( + "skipping target {:?} of kinds {:?}", + target.name, target.kind + ); + } + } + found.sort(); + found.dedup(); + Ok(found) +} + +fn should_mutate_target(target: &cargo_metadata::Target) -> bool { + target.kind.iter().any(|k| k.ends_with("lib") || k == "bin") +} + +/// Return the path of the workspace or package directory enclosing a given directory. +fn locate_project(path: &Utf8Path, workspace: bool) -> Result { + ensure!(path.is_dir(), "{path:?} is not a directory"); + let cargo_bin = cargo_bin(); // needed for lifetime + let mut argv: Vec<&str> = vec![&cargo_bin, "locate-project"]; + if workspace { + argv.push("--workspace"); + } + let stdout = get_command_output(&argv, path) + .with_context(|| format!("run cargo locate-project in {path:?}"))?; + let val: Value = serde_json::from_str(&stdout).context("parse cargo locate-project output")?; + let cargo_toml_path: Utf8PathBuf = val["root"] + .as_str() + .with_context(|| format!("cargo locate-project output has no root: {stdout:?}"))? + .to_owned() + .into(); + debug!(?cargo_toml_path, "Found workspace root manifest"); + ensure!( + cargo_toml_path.is_file(), + "cargo locate-project root {cargo_toml_path:?} is not a file" + ); + let root = cargo_toml_path + .parent() + .ok_or_else(|| anyhow!("cargo locate-project root {cargo_toml_path:?} has no parent"))? + .to_owned(); + ensure!( + root.is_dir(), + "apparent project root directory {root:?} is not a directory" + ); + Ok(root) +} + +#[cfg(test)] +mod test { + use std::ffi::OsStr; + + use camino::Utf8Path; + use itertools::Itertools; + + use crate::console::Console; + use crate::options::Options; + use crate::workspace::PackageFilter; + + use super::Workspace; + + #[test] + fn error_opening_outside_of_crate() { + Workspace::open("/").unwrap_err(); + } + + #[test] + fn open_subdirectory_of_crate_opens_the_crate() { + let workspace = Workspace::open("testdata/tree/factorial/src") + .expect("open source tree from subdirectory"); + let root = &workspace.dir; + assert!(root.is_dir()); + assert!(root.join("Cargo.toml").is_file()); + assert!(root.join("src/bin/factorial.rs").is_file()); + assert_eq!(root.file_name().unwrap(), OsStr::new("factorial")); + } + + #[test] + fn find_root_from_subdirectory_of_workspace_finds_the_workspace_root() { + let root = Workspace::open("testdata/tree/workspace/main") + .expect("Find root from within workspace/main") + .dir; + assert_eq!(root.file_name(), Some("workspace"), "Wrong root: {root:?}"); + } + + #[test] + fn find_top_source_files_from_subdirectory_of_workspace() { + let workspace = + Workspace::open("testdata/tree/workspace/main").expect("Find workspace root"); + assert_eq!( + workspace + .packages(&PackageFilter::All) + .unwrap() + .iter() + .map(|p| p.name.clone()) + .collect_vec(), + ["cargo_mutants_testdata_workspace_utils", "main", "main2"] + ); + assert_eq!( + workspace + .top_sources(&PackageFilter::All) + .unwrap() + .iter() + .map(|sf| sf.tree_relative_path.clone()) + .collect_vec(), + // ordered by package name + ["utils/src/lib.rs", "main/src/main.rs", "main2/src/main.rs"] + ); + } + + #[test] + fn package_filter_all_from_subdir_gets_everything() { + let subdir_path = Utf8Path::new("testdata/tree/workspace/main"); + let workspace = Workspace::open(subdir_path).expect("Find workspace root"); + let packages = workspace.packages(&PackageFilter::All).unwrap(); + assert_eq!( + packages.iter().map(|p| &p.name).collect_vec(), + ["cargo_mutants_testdata_workspace_utils", "main", "main2"] + ); + } + + #[test] + fn auto_packages_in_workspace_subdir_finds_single_package() { + let subdir_path = Utf8Path::new("testdata/tree/workspace/main"); + let workspace = Workspace::open(subdir_path).expect("Find workspace root"); + let packages = workspace + .packages(&PackageFilter::Auto(subdir_path.to_owned())) + .unwrap(); + assert_eq!(packages.iter().map(|p| &p.name).collect_vec(), ["main"]); + } + + #[test] + fn auto_packages_in_virtual_workspace_gets_everything() { + let path = Utf8Path::new("testdata/tree/workspace"); + let workspace = Workspace::open(path).expect("Find workspace root"); + let packages = workspace + .packages(&PackageFilter::Auto(path.to_owned())) + .unwrap(); + assert_eq!( + packages.iter().map(|p| &p.name).collect_vec(), + ["cargo_mutants_testdata_workspace_utils", "main", "main2"] + ); + } + + #[test] + fn filter_by_single_package() { + let workspace = + Workspace::open("testdata/tree/workspace/main").expect("Find workspace root"); + let root_dir = &workspace.dir; + assert_eq!( + root_dir.file_name(), + Some("workspace"), + "found the workspace root" + ); + let filter = PackageFilter::explicit(["main"]); + assert_eq!( + workspace + .packages(&filter) + .unwrap() + .iter() + .map(|p| p.name.clone()) + .collect_vec(), + ["main"] + ); + let top_sources = workspace.top_sources(&filter).unwrap(); + println!("{top_sources:#?}"); + assert_eq!( + top_sources + .iter() + .map(|sf| sf.tree_relative_path.clone()) + .collect_vec(), + [Utf8Path::new("main/src/main.rs")] + ); + } + + #[test] + fn filter_by_multiple_packages() { + let workspace = Workspace::open("testdata/tree/workspace/main").unwrap(); + assert_eq!( + workspace.dir.file_name(), + Some("workspace"), + "found the workspace root" + ); + let selection = PackageFilter::explicit(["main", "main2"]); + let discovered = workspace + .discover(&selection, &Options::default(), &Console::new()) + .unwrap(); + + assert_eq!( + discovered + .files + .iter() + .map(|sf| sf.tree_relative_path.clone()) + .collect_vec(), + ["main/src/main.rs", "main2/src/main.rs"] + ); + } +} diff --git a/tests/cli/config.rs b/tests/cli/config.rs index 29d1a3e3..e1a3870a 100644 --- a/tests/cli/config.rs +++ b/tests/cli/config.rs @@ -4,6 +4,7 @@ use std::fs::{create_dir, write}; +use indoc::indoc; use predicates::prelude::*; use tempfile::TempDir; @@ -87,10 +88,10 @@ fn list_with_config_file_inclusion() { .arg(testdata.path()) .assert() .success() - .stdout(predicates::str::diff( - "src/inside_mod.rs -src/item_mod.rs\n", - )); + .stdout(predicates::str::diff(indoc! { "\ + src/inside_mod.rs + src/item_mod.rs + " })); run() .args(["mutants", "--list", "-d"]) .arg(testdata.path()) diff --git a/tests/cli/main.rs b/tests/cli/main.rs index cc6cf084..7e3c5d1d 100644 --- a/tests/cli/main.rs +++ b/tests/cli/main.rs @@ -2,9 +2,10 @@ //! Tests for cargo-mutants CLI layer. +use std::borrow::Borrow; use std::env; use std::fmt::Write; -use std::fs::{self, read_dir, read_to_string}; +use std::fs::{self, read_dir}; use std::io::Read; use std::path::{Path, PathBuf}; use std::thread::sleep; @@ -27,6 +28,7 @@ mod jobs; mod trace; #[cfg(windows)] mod windows; +mod workspace; /// A timeout for a `cargo mutants` invocation from the test suite. Needs to be /// long enough that even commands that do a lot of work can pass even on slow @@ -96,6 +98,17 @@ fn redact_timestamps_sizes(s: &str) -> String { SIZE_RE.replace_all(&s, "xxx MB").to_string() } +/// Assert that some bytes, when parsed as json, equal a json value. +fn assert_bytes_eq_json>(actual: &[u8], expected: J) { + // The Borrow is so that you can pass either a value or a reference, for easier + // calling. + let actual_json = std::str::from_utf8(actual) + .expect("bytes are UTF-8") + .parse::() + .expect("bytes can be parsed as JSON"); + assert_eq!(&actual_json, expected.borrow()); +} + #[test] fn incorrect_cargo_subcommand() { // argv[1] "mutants" is missing here. @@ -462,157 +475,6 @@ fn list_files_json_well_tested() { .current_dir("testdata/tree/well_tested") .assert_insta("list_files_json_well_tested"); } - -#[test] -fn list_files_json_workspace() { - // Demonstrates that we get package names in the json listing. - run() - .args(["mutants", "--list-files", "--json"]) - .current_dir("testdata/tree/workspace") - .assert_insta("list_files_json_workspace"); -} - -#[test] -fn list_files_as_json_in_workspace_subdir() { - run() - .args(["mutants", "--list-files", "--json"]) - .current_dir("testdata/tree/workspace/main2") - .assert() - .stdout(indoc! {r#" - [ - { - "package": "cargo_mutants_testdata_workspace_utils", - "path": "utils/src/lib.rs" - }, - { - "package": "main", - "path": "main/src/main.rs" - }, - { - "package": "main2", - "path": "main2/src/main.rs" - } - ] - "#}); -} - -#[test] -fn workspace_tree_is_well_tested() { - let tmp_src_dir = copy_of_testdata("workspace"); - run() - .args(["mutants", "-d"]) - .arg(tmp_src_dir.path()) - .assert() - .success(); - // The outcomes.json has some summary data - let json_str = - fs::read_to_string(tmp_src_dir.path().join("mutants.out/outcomes.json")).unwrap(); - println!("outcomes.json:\n{json_str}"); - let json: serde_json::Value = json_str.parse().unwrap(); - assert_eq!(json["total_mutants"].as_u64().unwrap(), 8); - assert_eq!(json["caught"].as_u64().unwrap(), 8); - assert_eq!(json["missed"].as_u64().unwrap(), 0); - assert_eq!(json["timeout"].as_u64().unwrap(), 0); - let outcomes = json["outcomes"].as_array().unwrap(); - - { - let baseline = outcomes[0].as_object().unwrap(); - assert_eq!(baseline["scenario"].as_str().unwrap(), "Baseline"); - assert_eq!(baseline["summary"], "Success"); - let baseline_phases = baseline["phase_results"].as_array().unwrap(); - assert_eq!(baseline_phases.len(), 2); - assert_eq!(baseline_phases[0]["process_status"], "Success"); - assert_eq!( - baseline_phases[0]["argv"].as_array().unwrap().iter().map(|v| v.as_str().unwrap()).skip(1).collect_vec().join(" "), - "build --tests --package cargo_mutants_testdata_workspace_utils --package main --package main2" - ); - assert_eq!(baseline_phases[1]["process_status"], "Success"); - assert_eq!( - baseline_phases[1]["argv"] - .as_array() - .unwrap() - .iter() - .map(|v| v.as_str().unwrap()) - .skip(1) - .collect_vec() - .join(" "), - "test --package cargo_mutants_testdata_workspace_utils --package main --package main2" - ); - } - - assert_eq!(outcomes.len(), 9); - for outcome in &outcomes[1..] { - let mutant = &outcome["scenario"]["Mutant"]; - let package_name = mutant["package"].as_str().unwrap(); - assert!(!package_name.is_empty()); - assert_eq!(outcome["summary"], "CaughtMutant"); - let mutant_phases = outcome["phase_results"].as_array().unwrap(); - assert_eq!(mutant_phases.len(), 2); - assert_eq!(mutant_phases[0]["process_status"], "Success"); - assert_eq!( - mutant_phases[0]["argv"].as_array().unwrap()[1..=3], - ["build", "--tests", "--manifest-path"] - ); - assert_eq!(mutant_phases[1]["process_status"], "Failure"); - assert_eq!( - mutant_phases[1]["argv"].as_array().unwrap()[1..=2], - ["test", "--manifest-path"], - ); - } - { - let baseline = json["outcomes"][0].as_object().unwrap(); - assert_eq!(baseline["scenario"].as_str().unwrap(), "Baseline"); - assert_eq!(baseline["summary"], "Success"); - let baseline_phases = baseline["phase_results"].as_array().unwrap(); - assert_eq!(baseline_phases.len(), 2); - assert_eq!(baseline_phases[0]["process_status"], "Success"); - assert_eq!( - baseline_phases[0]["argv"].as_array().unwrap()[1..].iter().map(|v| v.as_str().unwrap()).join(" "), - "build --tests --package cargo_mutants_testdata_workspace_utils --package main --package main2", - ); - assert_eq!(baseline_phases[1]["process_status"], "Success"); - assert_eq!( - baseline_phases[1]["argv"].as_array().unwrap()[1..] - .iter() - .map(|v| v.as_str().unwrap()) - .join(" "), - "test --package cargo_mutants_testdata_workspace_utils --package main --package main2", - ); - } -} - -#[test] -/// Baseline tests in a workspace only test the packages that will later -/// be mutated. -/// See -fn in_workspace_only_relevant_packages_included_in_baseline_tests() { - let tmp = copy_of_testdata("package_fails"); - run() - .args(["mutants", "-f", "passing/src/lib.rs", "--no-shuffle", "-d"]) - .arg(tmp.path()) - .assert() - .success(); - assert_eq!( - read_to_string(tmp.path().join("mutants.out/caught.txt")).unwrap(), - indoc! { "\ - passing/src/lib.rs:1: replace triple -> usize with 0 - passing/src/lib.rs:1: replace triple -> usize with 1 - "} - ); - assert_eq!( - read_to_string(tmp.path().join("mutants.out/timeout.txt")).unwrap(), - "" - ); - assert_eq!( - read_to_string(tmp.path().join("mutants.out/missed.txt")).unwrap(), - "" - ); - assert_eq!( - read_to_string(tmp.path().join("mutants.out/unviable.txt")).unwrap(), - "" - ); -} - #[test] fn copy_testdata_doesnt_include_build_artifacts() { // If there is a target or mutants.out in the source directory, we don't want it in the copy, diff --git a/tests/cli/workspace.rs b/tests/cli/workspace.rs new file mode 100644 index 00000000..79b4e3d5 --- /dev/null +++ b/tests/cli/workspace.rs @@ -0,0 +1,240 @@ +// Copyright 2023 Martin Pool + +//! Tests for cargo workspaces with multiple packages. + +use std::fs::{self, read_to_string}; + +use indoc::indoc; +use itertools::Itertools; +use serde_json::json; + +use super::{assert_bytes_eq_json, copy_of_testdata, run}; + +#[test] +fn list_warns_about_unmatched_packages() { + run() + .args([ + "mutants", + "--list", + "-d", + "testdata/tree/workspace", + "-p", + "notapackage", + ]) + .assert() + .stderr(predicates::str::contains( + "package \"notapackage\" not found in source tree", + )) + .code(0); +} + +#[test] +fn list_files_json_workspace() { + // Demonstrates that we get package names in the json listing. + let cmd = run() + .args(["mutants", "--list-files", "--json"]) + .current_dir("testdata/tree/workspace") + .assert() + .success(); + assert_bytes_eq_json( + &cmd.get_output().stdout, + json! { + [ + { + "package": "cargo_mutants_testdata_workspace_utils", + "path": "utils/src/lib.rs" + }, + { + "package": "main", + "path": "main/src/main.rs" + }, + { + "package": "main2", + "path": "main2/src/main.rs" + } + ] + }, + ); +} + +#[test] +fn list_files_as_json_in_workspace_subdir() { + let cmd = run() + .args(["mutants", "--list-files", "--json", "--workspace"]) + .current_dir("testdata/tree/workspace/main2") + .assert() + .success(); + assert_bytes_eq_json( + &cmd.get_output().stdout, + json! { + [ + { + "package": "cargo_mutants_testdata_workspace_utils", + "path": "utils/src/lib.rs" + }, + { + "package": "main", + "path": "main/src/main.rs" + }, + { + "package": "main2", + "path": "main2/src/main.rs" + } + ] + }, + ); +} + +#[test] +fn workspace_tree_is_well_tested() { + let tmp_src_dir = copy_of_testdata("workspace"); + run() + .args(["mutants", "-d"]) + .arg(tmp_src_dir.path()) + .assert() + .success(); + // The outcomes.json has some summary data + let json_str = + fs::read_to_string(tmp_src_dir.path().join("mutants.out/outcomes.json")).unwrap(); + println!("outcomes.json:\n{json_str}"); + let json: serde_json::Value = json_str.parse().unwrap(); + assert_eq!(json["total_mutants"].as_u64().unwrap(), 8); + assert_eq!(json["caught"].as_u64().unwrap(), 8); + assert_eq!(json["missed"].as_u64().unwrap(), 0); + assert_eq!(json["timeout"].as_u64().unwrap(), 0); + let outcomes = json["outcomes"].as_array().unwrap(); + + { + let baseline = outcomes[0].as_object().unwrap(); + assert_eq!(baseline["scenario"].as_str().unwrap(), "Baseline"); + assert_eq!(baseline["summary"], "Success"); + let baseline_phases = baseline["phase_results"].as_array().unwrap(); + assert_eq!(baseline_phases.len(), 2); + assert_eq!(baseline_phases[0]["process_status"], "Success"); + assert_eq!( + baseline_phases[0]["argv"].as_array().unwrap().iter().map(|v| v.as_str().unwrap()).skip(1).collect_vec().join(" "), + "build --tests --package cargo_mutants_testdata_workspace_utils --package main --package main2" + ); + assert_eq!(baseline_phases[1]["process_status"], "Success"); + assert_eq!( + baseline_phases[1]["argv"] + .as_array() + .unwrap() + .iter() + .map(|v| v.as_str().unwrap()) + .skip(1) + .collect_vec() + .join(" "), + "test --package cargo_mutants_testdata_workspace_utils --package main --package main2" + ); + } + + assert_eq!(outcomes.len(), 9); + for outcome in &outcomes[1..] { + let mutant = &outcome["scenario"]["Mutant"]; + let package_name = mutant["package"].as_str().unwrap(); + assert!(!package_name.is_empty()); + assert_eq!(outcome["summary"], "CaughtMutant"); + let mutant_phases = outcome["phase_results"].as_array().unwrap(); + assert_eq!(mutant_phases.len(), 2); + assert_eq!(mutant_phases[0]["process_status"], "Success"); + assert_eq!( + mutant_phases[0]["argv"].as_array().unwrap()[1..=3], + ["build", "--tests", "--manifest-path"] + ); + assert_eq!(mutant_phases[1]["process_status"], "Failure"); + assert_eq!( + mutant_phases[1]["argv"].as_array().unwrap()[1..=2], + ["test", "--manifest-path"], + ); + } + { + let baseline = json["outcomes"][0].as_object().unwrap(); + assert_eq!(baseline["scenario"].as_str().unwrap(), "Baseline"); + assert_eq!(baseline["summary"], "Success"); + let baseline_phases = baseline["phase_results"].as_array().unwrap(); + assert_eq!(baseline_phases.len(), 2); + assert_eq!(baseline_phases[0]["process_status"], "Success"); + assert_eq!( + baseline_phases[0]["argv"].as_array().unwrap()[1..].iter().map(|v| v.as_str().unwrap()).join(" "), + "build --tests --package cargo_mutants_testdata_workspace_utils --package main --package main2", + ); + assert_eq!(baseline_phases[1]["process_status"], "Success"); + assert_eq!( + baseline_phases[1]["argv"].as_array().unwrap()[1..] + .iter() + .map(|v| v.as_str().unwrap()) + .join(" "), + "test --package cargo_mutants_testdata_workspace_utils --package main --package main2", + ); + } +} + +#[test] +/// Baseline tests in a workspace only test the packages that will later +/// be mutated. +/// See +fn in_workspace_only_relevant_packages_included_in_baseline_tests_by_file_filter() { + let tmp = copy_of_testdata("package_fails"); + run() + .args(["mutants", "-f", "passing/src/lib.rs", "--no-shuffle", "-d"]) + .arg(tmp.path()) + .assert() + .success(); + assert_eq!( + read_to_string(tmp.path().join("mutants.out/caught.txt")).unwrap(), + indoc! { "\ + passing/src/lib.rs:1: replace triple -> usize with 0 + passing/src/lib.rs:1: replace triple -> usize with 1 + "} + ); + assert_eq!( + read_to_string(tmp.path().join("mutants.out/timeout.txt")).unwrap(), + "" + ); + assert_eq!( + read_to_string(tmp.path().join("mutants.out/missed.txt")).unwrap(), + "" + ); + assert_eq!( + read_to_string(tmp.path().join("mutants.out/unviable.txt")).unwrap(), + "" + ); +} + +/// Even the baseline test only tests the explicitly selected packages, +/// so it doesn't fail if some packages don't build. +#[test] +fn baseline_test_respects_package_options() { + let tmp = copy_of_testdata("package_fails"); + run() + .args([ + "mutants", + "--package", + "cargo-mutants-testdata-package-fails-passing", + "--no-shuffle", + "-d", + ]) + .arg(tmp.path()) + .assert() + .success(); + assert_eq!( + read_to_string(tmp.path().join("mutants.out/caught.txt")).unwrap(), + indoc! { "\ + passing/src/lib.rs:1: replace triple -> usize with 0 + passing/src/lib.rs:1: replace triple -> usize with 1 + "} + ); + assert_eq!( + read_to_string(tmp.path().join("mutants.out/timeout.txt")).unwrap(), + "" + ); + assert_eq!( + read_to_string(tmp.path().join("mutants.out/missed.txt")).unwrap(), + "" + ); + assert_eq!( + read_to_string(tmp.path().join("mutants.out/unviable.txt")).unwrap(), + "" + ); +}