diff --git a/.cargo/mutants.toml b/.cargo/mutants.toml new file mode 100644 index 00000000..b5ea9aa8 --- /dev/null +++ b/.cargo/mutants.toml @@ -0,0 +1,3 @@ +# cargo-mutants configuration + +exclude_globs = [ "src/console.rs" ] diff --git a/Cargo.toml b/Cargo.toml index 68eb0f8a..9fcac345 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -85,6 +85,7 @@ exclude = [ "testdata/tree/dependency", "testdata/tree/everything_skipped", "testdata/tree/factorial", + "testdata/tree/fails_without_feature", "testdata/tree/hang_avoided_by_attr/", "testdata/tree/hang_when_mutated", "testdata/tree/insta", diff --git a/NEWS.md b/NEWS.md index e7af86bf..f5ba922d 100644 --- a/NEWS.md +++ b/NEWS.md @@ -4,6 +4,12 @@ - Fixed: Files that are excluded by filters are also excluded from `--list-files`. +- Fixed: `--exclude-re` and `--re` can match against the return type as shown in + `--list`. + +- New: A `.cargo/mutants.toml` file can be used to configure standard filters + and cargo args for a project. + ## 1.1.1 Released 2022-10-31 diff --git a/README.md b/README.md index b55a04ac..31a696ed 100644 --- a/README.md +++ b/README.md @@ -179,19 +179,19 @@ passing when run from a different location, such as a relative `path` in Otherwise, cargo mutants generates every mutant it can. All mutants fall in to one of these categories: -- **caught** — A test failed with this mutant applied. This is a good sign about +* **caught** — A test failed with this mutant applied. This is a good sign about test coverage. You can look in `mutants.out/log` to see which tests failed. -- **missed** — No test failed with this mutation applied, which seems to +* **missed** — No test failed with this mutation applied, which seems to indicate a gap in test coverage. Or, it may be that the mutant is undistinguishable from the correct code. You may wish to add a better test, or mark that the function should be skipped. -- **unviable** — The attempted mutation doesn't compile. This is inconclusive about test coverage and +* **unviable** — The attempted mutation doesn't compile. This is inconclusive about test coverage and no action is needed, but indicates an opportunity for cargo-mutants to either generate better mutants, or at least not generate unviable mutants. -- **timeout** — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and potentially mark the function to be skipped. +* **timeout** — The mutation caused the test suite to run for a long time, until it was eventually killed. You might want to investigate the cause and potentially mark the function to be skipped. By default only missed mutants and timeouts are printed because they're the most actionable. Others can be shown with the `-v` and `-V` options. @@ -214,39 +214,63 @@ flags the function for cargo-mutants. **Note:** Rust's "inner macro attributes" feature is currently unstable, so `#![mutants::skip]` can't be used at the top of a file in stable Rust. +### Config file + +cargo-mutants looks for a `.cargo/mutants.toml` file in the root of the source +directory. If a config file exists, the values are appended to the corresponding +command-line arguments. (This may cause problems if you use `--` twice on the +command line to pass arguments to the inner test binary.) + +Configured exclusions may be particularly important when there are modules that +are inherently hard to test, and the project has made a decision to accept lower +test coverage for them. + +The following configuration options are supported: + +```toml +exclude_globs = ["src/main.rs", "src/cache/*.rs"] # same as -e +examine_globs = ["src/important/*.rs"] # same as -f, test *only* these files + +exclude_re = ["impl Debug"] # same as -E +examine_re = ["impl Serialize", "impl Deserialize"] # same as -F, test *only* matches + +additional_cargo_args = ["--all-features"] +additional_cargo_test_args = ["--jobs=1"] +``` + ### Exit codes -- **0**: Success. No mutants were found that weren't caught by tests. +* **0**: Success. No mutants were found that weren't caught by tests. -- **1**: Usage error: bad command-line arguments etc. +* **1**: Usage error: bad command-line arguments etc. -- **2**: Found some mutants that were not covered by tests. +* **2**: Found some mutants that were not covered by tests. -- **3**: Some tests timed out: possibly the mutatations caused an infinite loop, +* **3**: Some tests timed out: possibly the mutatations caused an infinite loop, or the timeout is too low. -- **4**: The tests are already failing or hanging before any mutations are +* **4**: The tests are already failing or hanging before any mutations are applied, so no mutations were tested. ### `mutants.out` A `mutants.out` directory is created in the source directory, or whichever directory you specify with `--output`. It contains: -- A `logs/` directory, with one log file for each mutation plus the baseline +* A `logs/` directory, with one log file for each mutation plus the baseline unmutated case. The log contains the diff of the mutation plus the output from cargo. -- A `lock.json`, on which an [fs2 lock](https://docs.rs/fs2) is held while +* A `lock.json`, on which an [fs2 lock](https://docs.rs/fs2) is held while cargo-mutants is running, to avoid two tasks trying to write to the same directory at the same time. The lock contains the start time, cargo-mutants version, username, and hostname. `lock.json` is left in `mutants.out` when the run completes, but the lock on it is released. -- `caught.txt`, `missed.txt`, `timeout.txt`, `unviable.txt`, each listing mutants with the corresponding outcome. +* `caught.txt`, `missed.txt`, `timeout.txt`, `unviable.txt`, each listing mutants with the corresponding outcome. -- A `mutants.json` file describing all the generated mutants. +* A `mutants.json` file describing all the generated mutants. -- An `outcomes.json` file describing the results of all tests. +* An `outcomes.json` file describing the results of all tests. ### Hangs and timeouts @@ -342,12 +366,12 @@ performance side effects. Ideally, these should be tested, but doing so in a way that's not flaky can be difficult. cargo-mutants can help in a few ways: -- It helps to at least highlight to the developer that the function is not +* It helps to at least highlight to the developer that the function is not covered by tests, and so should perhaps be treated with extra care, or tested manually. -- A `#[mutants::skip]` annotation can be added to suppress warnings and explain +* A `#[mutants::skip]` annotation can be added to suppress warnings and explain the decision. -- Sometimes these effects can be tested by making the side-effect observable +* Sometimes these effects can be tested by making the side-effect observable with, for example, a counter of the number of memory allocations or cache misses/hits. @@ -384,8 +408,8 @@ jobs: Experience reports in [GitHub Discussions](https://github.com/sourcefrog/cargo-mutants/discussions) or issues are very welcome: -- Did it find a bug or important coverage gap? -- Did it fail to build and test your tree? (Some cases that aren't supported yet +* Did it find a bug or important coverage gap? +* Did it fail to build and test your tree? (Some cases that aren't supported yet are already listed in this doc or the bug tracker.) It's especially helpful if you can either point to an open source tree that will @@ -401,16 +425,16 @@ the tests might be insufficient.** Being _easy_ to use means: -- cargo-mutants requires no changes to the source tree or other setup: just +* cargo-mutants requires no changes to the source tree or other setup: just install and run. So, if it does not find anything interesting to say about a well-tested tree, it didn't cost you much. (This worked out really well: `cargo install cargo-mutants && cargo mutants` will do it.) -- There is no chance that running cargo-mutants will change the released +* There is no chance that running cargo-mutants will change the released behavior of your program (other than by helping you to fix bugs!), because you don't need to change the source to use it. -- cargo-mutants should be reasonably fast even on large Rust trees. The overall +* cargo-mutants should be reasonably fast even on large Rust trees. The overall run time is, roughly, the product of the number of viable mutations multiplied by the time to run the test suite for each mutation. Typically, one `cargo mutants` run will give you all the information it can find about missing test @@ -420,24 +444,24 @@ Being _easy_ to use means: for each mutant, but that can still be significant for large trees. There's room to improve by testing multiple mutants in parallel.) -- cargo-mutants should run correctly on any Rust source trees that are built and +* cargo-mutants should run correctly on any Rust source trees that are built and tested by Cargo, that will build and run their tests in a copy of the tree, and that have hermetic tests. (It's not all the way there yet; in particular it assumes the source is in `src/`.) -- cargo-mutants shouldn't crash or hang, even if it generates mutants that cause +* cargo-mutants shouldn't crash or hang, even if it generates mutants that cause the software under test to crash or hang. (This is generally met today: cargo-mutants runs tests with an automatically set and configurable timeout.) -- The results should be reproducible, assuming the build and test suite is +* The results should be reproducible, assuming the build and test suite is deterministic. (This should be true today; please file a bug if it's not. Mutants are run in random order unless `--no-shuffle` is specified, but this should not affect the results.) -- cargo-mutants should avoid generating unviable mutants that don't compile, +* cargo-mutants should avoid generating unviable mutants that don't compile, because that wastes time. However, when it's uncertain whether the mutant will build, it's worth trying things that _might_ find interesting results even if they might fail to build. (It does currently generate _some_ unviable mutants, but typically not too many, and they don't have a large effect on runtime in most trees.) -- Realistically, cargo-mutants may generate some mutants that aren't caught by +* Realistically, cargo-mutants may generate some mutants that aren't caught by tests but also aren't interesting, or aren't feasible to test. In those cases it should be easy to permanently dismiss them (e.g. by adding a `#[mutants::skip]` attribute or a config file.) (The attribute exists but @@ -445,66 +469,66 @@ Being _easy_ to use means: Showing _interesting results_ mean: -- cargo-mutants should tell you about places where the code could be wrong and +* cargo-mutants should tell you about places where the code could be wrong and the test suite wouldn't catch it. If it doesn't find any interesting results on typical trees, there's no point. Aspirationally, it will even find useful results in code with high line coverage, when there is code that is reached by a test, but no test depends on its behavior. -- In superbly-tested projects cargo-mutants may find nothing to say, but hey, at +* In superbly-tested projects cargo-mutants may find nothing to say, but hey, at least it was easy to run, and hopefully the assurance that the tests really do seem to be good is useful data. -- _Most_, ideally all, findings should indicate something that really should be +* _Most_, ideally all, findings should indicate something that really should be tested more, or that may already be buggy, or that's at least worth looking at. -- It should be easy to understand what the output is telling you about a +* It should be easy to understand what the output is telling you about a potential bug that wouldn't be caught. (This seems true today.) It might take some thought to work out _why_ the existing tests don't cover it, or how to check it, but at least you know where to begin. -- As much as possible cargo-mutants should avoid generating trivial mutants, +* As much as possible cargo-mutants should avoid generating trivial mutants, where the mutated code is effectively equivalent to the original code, and so it's not interesting that the test suite doesn't catch the change. (Not much has been done here yet.) -- For trees that are thoroughly tested, you can use `cargo mutants` in CI to +* For trees that are thoroughly tested, you can use `cargo mutants` in CI to check that they remain so. ## How it works The basic approach is: -- Make a copy of the source tree into a scratch directory, excluding +* Make a copy of the source tree into a scratch directory, excluding version-control directories like `.git` and the `/target` directory. The same directory is reused across all the mutations to benefit from incremental builds. - - After copying the tree, cargo-mutants scans the top-level `Cargo.toml` and any + * After copying the tree, cargo-mutants scans the top-level `Cargo.toml` and any `.cargo/config.toml` for relative dependencies. If there are any, the paths are rewritten to be absolute, so that they still work when cargo is run in the scratch directory. - - Before applying any mutations, check that `cargo test` succeeds in the + * Before applying any mutations, check that `cargo test` succeeds in the scratch directory: perhaps a test is already broken, or perhaps the tree doesn't build when copied because it relies on relative paths to find dependencies, etc. -- Build a list of mutations: - - Run `cargo metadata` to find directories containing Rust source files. - - Walk all source files and parse each one looking for functions. - - Skip functions that should not be mutated for any of several reasons: +* Build a list of mutations: + * Run `cargo metadata` to find directories containing Rust source files. + * Walk all source files and parse each one looking for functions. + * Skip functions that should not be mutated for any of several reasons: because they're tests, because they have a `#[mutants::skip]` attribute, etc. - - For each function, depending on its return type, generate every mutation + * For each function, depending on its return type, generate every mutation pattern that produces a result of that type. -- For each mutation: - - Apply the mutation to the scratch tree by patching the affected file. - - Run `cargo test` in the tree, saving output to a log file. - - If the build fails or the tests fail, that's good: the mutation was somehow +* For each mutation: + * Apply the mutation to the scratch tree by patching the affected file. + * Run `cargo test` in the tree, saving output to a log file. + * If the build fails or the tests fail, that's good: the mutation was somehow caught. - - If the build and tests succeed, that might mean test coverage was + * If the build and tests succeed, that might mean test coverage was inadequate, or it might mean we accidentally generated a no-op mutation. - - Revert the mutation to return the tree to its clean state. + * Revert the mutation to return the tree to its clean state. The file is parsed using the [`syn`](https://docs.rs/syn) crate, but mutations are applied textually, rather than to the token stream, so that unmutated code diff --git a/src/config.rs b/src/config.rs new file mode 100644 index 00000000..d31a38a3 --- /dev/null +++ b/src/config.rs @@ -0,0 +1,58 @@ +// Copyright 2022 Martin Pool. + +//! `.cargo/mutants.toml` configuration file. +//! +//! The config file is read after parsing command line arguments, +//! and after finding the source tree, because these together +//! determine its location. +//! +//! The config file is then merged in to the [Options]. + +use std::default::Default; +use std::fs::read_to_string; + +use anyhow::Context; +use camino::Utf8Path; +use serde::Deserialize; + +use crate::source::SourceTree; +use crate::Result; + +/// Configuration read from a config file. +/// +/// This is similar to [Options], and eventually merged into it, but separate because it +/// can be deserialized. +#[derive(Debug, Default, Clone, PartialEq, Eq, Deserialize)] +#[serde(default, deny_unknown_fields)] +pub struct Config { + /// Generate mutants from source files matching these globs. + pub examine_globs: Vec, + /// Exclude mutants from source files matching these globs. + pub exclude_globs: Vec, + /// Exclude mutants from source files matches these regexps. + pub exclude_re: Vec, + /// Examine only mutants matching these regexps. + pub examine_re: Vec, + /// Pass extra args to every cargo invocation. + pub additional_cargo_args: Vec, + /// Pass extra args to cargo test. + pub additional_cargo_test_args: Vec, +} + +impl Config { + pub fn read_file(path: &Utf8Path) -> Result { + let toml = read_to_string(path).with_context(|| format!("read config {path:?}"))?; + toml::de::from_str(&toml).with_context(|| format!("parse toml from {path:?}")) + } + + /// Read the config from a tree's `.cargo/mutants.toml`, and return a default (empty) + /// Config is the file does not exist. + pub fn read_tree_config(source_tree: &dyn SourceTree) -> Result { + let path = source_tree.path().join(".cargo").join("mutants.toml"); + if path.exists() { + Config::read_file(&path) + } else { + Ok(Config::default()) + } + } +} diff --git a/src/console.rs b/src/console.rs index 285007c0..018a7cce 100644 --- a/src/console.rs +++ b/src/console.rs @@ -540,6 +540,8 @@ pub fn list_mutants(mutants: &[Mutant], show_diffs: bool) { } fn style_mutant(mutant: &Mutant) -> String { + // This is like `impl Display for Mutant`, but with colors. + // The text content should be the same. format!( "{}: replace {}{}{} with {}", mutant.describe_location(), diff --git a/src/main.rs b/src/main.rs index 5d197b10..a897c04d 100644 --- a/src/main.rs +++ b/src/main.rs @@ -4,6 +4,7 @@ mod build_dir; mod cargo; +mod config; mod console; mod exit_code; mod interrupt; @@ -21,7 +22,6 @@ mod source; mod textedit; mod visit; -use std::convert::TryFrom; use std::env; use std::io::{self, Write}; use std::process::exit; @@ -35,6 +35,7 @@ use clap::Parser; use clap_complete::{generate, Shell}; use path_slash::PathExt; use serde_json::{json, Value}; +use tracing::debug; // Imports of public names from this crate. use crate::build_dir::BuildDir; @@ -186,9 +187,6 @@ fn main() -> Result<()> { console.setup_global_trace(args.level)?; interrupt::install_handler(); - let options = Options::try_from(&args)?; - // dbg!(&options); - if args.version { println!("{} {}", NAME, VERSION); return Ok(()); @@ -197,8 +195,16 @@ fn main() -> Result<()> { return Ok(()); } - let source_path = args.dir.unwrap_or_else(|| Utf8Path::new(".").to_owned()); - let source_tree = CargoSourceTree::open(&source_path)?; + let source_path: &Utf8Path = if let Some(p) = &args.dir { + p + } else { + Utf8Path::new(".") + }; + let source_tree = CargoSourceTree::open(source_path)?; + let config = config::Config::read_tree_config(&source_tree)?; + debug!(?config); + let options = Options::new(&args, &config)?; + debug!(?options); if args.list_files { list_files(&source_tree, &options, args.json)?; } else if args.list { diff --git a/src/mutate.rs b/src/mutate.rs index 0866f64e..776f5ce5 100644 --- a/src/mutate.rs +++ b/src/mutate.rs @@ -127,12 +127,18 @@ impl Mutant { /// Describe the mutant briefly, not including the location. /// - /// The result is like `replace source::SourceFile::new with Default::default()`. + /// The result is like `replace factorial -> u32 with Default::default()`. pub fn describe_change(&self) -> String { format!( - "replace {} with {}", - self.function_name(), - self.op.replacement() + "replace {name}{space}{type} with {replacement}", + name = self.function_name(), + space = if self.return_type.is_empty() { + "" + } else { + " " + }, + type = self.return_type(), + replacement = self.op.replacement() ) } @@ -213,12 +219,14 @@ impl fmt::Display for Mutant { /// /// The result is like `src/source.rs:123: replace source::SourceFile::new with Default::default()`. fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + // This is like `style_mutant`, but without colors. + // The text content should be the same. write!( f, - "{}:{}: {}", - self.source_file.tree_relative_slashes(), - self.span.start.line, - self.describe_change() + "{file}:{line}: {change}", + file = self.source_file.tree_relative_slashes(), + line = self.span.start.line, + change = self.describe_change() ) } } @@ -269,7 +277,7 @@ mod test { ); assert_eq!( mutants[1].to_string(), - "src/bin/factorial.rs:7: replace factorial with Default::default()" + "src/bin/factorial.rs:7: replace factorial -> u32 with Default::default()" ); } diff --git a/src/options.rs b/src/options.rs index 99eec420..e1546ad1 100644 --- a/src/options.rs +++ b/src/options.rs @@ -4,7 +4,6 @@ //! //! The [Options] structure is built from command-line options and then widely passed around. -use std::convert::TryFrom; use std::time::Duration; use anyhow::Context; @@ -13,7 +12,7 @@ use globset::{Glob, GlobSet, GlobSetBuilder}; use regex::RegexSet; use tracing::warn; -use crate::*; +use crate::{config::Config, *}; /// Options for running experiments. #[derive(Default, Debug, Clone)] @@ -63,29 +62,43 @@ pub struct Options { /// Create `mutants.out` within this directory (by default, the source directory). pub output_in_dir: Option, + /// Run this many `cargo build` or `cargo test` tasks in parallel. pub jobs: Option, } -impl TryFrom<&Args> for Options { - type Error = anyhow::Error; - - fn try_from(args: &Args) -> Result { +impl Options { + /// Build options by merging command-line args and config file. + pub(crate) fn new(args: &Args, config: &Config) -> Result { if args.no_copy_target { warn!("--no-copy-target is deprecated and has no effect; target/ is never copied"); } Ok(Options { - additional_cargo_args: args.cargo_arg.clone(), - additional_cargo_test_args: args.cargo_test_args.clone(), + additional_cargo_args: args + .cargo_arg + .iter() + .cloned() + .chain(config.additional_cargo_args.iter().cloned()) + .collect(), + additional_cargo_test_args: args + .cargo_test_args + .iter() + .cloned() + .chain(config.additional_cargo_test_args.iter().cloned()) + .collect(), check_only: args.check, examine_names: Some( - RegexSet::new(&args.examine_re).context("Compiling examine_re regex")?, + RegexSet::new(args.examine_re.iter().chain(config.examine_re.iter())) + .context("Compiling examine_re regex")?, ), - examine_globset: build_glob_set(&args.file)?, + examine_globset: build_glob_set(args.file.iter().chain(config.examine_globs.iter()))?, exclude_names: Some( - RegexSet::new(&args.exclude_re).context("Compiling exclude_re regex")?, + RegexSet::new(args.exclude_re.iter().chain(config.exclude_re.iter())) + .context("Compiling exclude_re regex")?, ), - exclude_globset: build_glob_set(&args.exclude)?, + exclude_globset: build_glob_set( + args.exclude.iter().chain(config.exclude_globs.iter()), + )?, jobs: args.jobs, output_in_dir: args.output.clone(), print_caught: args.caught, @@ -98,13 +111,17 @@ impl TryFrom<&Args> for Options { } } -fn build_glob_set(glob_set: &Vec) -> Result> { - if glob_set.is_empty() { +fn build_glob_set, I: IntoIterator>( + glob_set: I, +) -> Result> { + let mut glob_set = glob_set.into_iter().peekable(); + if glob_set.peek().is_none() { return Ok(None); } let mut builder = GlobSetBuilder::new(); for glob_str in glob_set { + let glob_str = glob_str.as_ref(); if glob_str.contains('/') || glob_str.contains(std::path::MAIN_SEPARATOR) { builder.add(Glob::new(glob_str)?); } else { diff --git a/src/visit.rs b/src/visit.rs index 877df59c..2fae590d 100644 --- a/src/visit.rs +++ b/src/visit.rs @@ -10,10 +10,8 @@ use std::sync::Arc; use anyhow::Context; use quote::ToTokens; use syn::visit::Visit; -use syn::Attribute; -use syn::ItemFn; -use tracing::warn; -use tracing::{debug, debug_span, span, trace, Level}; +use syn::{Attribute, ItemFn}; +use tracing::{debug, debug_span, span, trace, warn, Level}; use crate::path::TreeRelativePathBuf; use crate::source::{SourceFile, SourceTree}; diff --git a/testdata/tree/fails_without_feature/Cargo.toml b/testdata/tree/fails_without_feature/Cargo.toml new file mode 100644 index 00000000..18011329 --- /dev/null +++ b/testdata/tree/fails_without_feature/Cargo.toml @@ -0,0 +1,16 @@ +[package] +name = "cargo-mutants-testdata-fails-without-feature" +version = "0.0.0" +edition = "2021" +authors = ["Martin Pool"] +publish = false + +[dependencies.mutants] +version = "0.0.3" + +[features] +needed = [] + +[[bin]] +name = "factorial" +doctest = false diff --git a/testdata/tree/fails_without_feature/README.md b/testdata/tree/fails_without_feature/README.md new file mode 100644 index 00000000..581086f6 --- /dev/null +++ b/testdata/tree/fails_without_feature/README.md @@ -0,0 +1,5 @@ +# `fails_without_feature` + +The crates for this test fail unless a feature is turned on. + +(Not a good style perhaps, but a good way to test that Cargo features can be passed through.) diff --git a/testdata/tree/fails_without_feature/src/bin/factorial.rs b/testdata/tree/fails_without_feature/src/bin/factorial.rs new file mode 100644 index 00000000..6051cb23 --- /dev/null +++ b/testdata/tree/fails_without_feature/src/bin/factorial.rs @@ -0,0 +1,27 @@ +#[mutants::skip] +fn main() { + for i in 1..=6 { + println!("{}! = {}", i, factorial(i)); + } +} + +#[cfg(feature = "needed")] +fn factorial(n: u32) -> u32 { + let mut a = 1; + for i in 2..=n { + a *= i; + } + a +} + +#[cfg(not(feature = "needed"))] +#[mutants::skip] +fn factorial(_n: u32) -> u32 { + panic!("needed feature is not enabled"); +} + +#[test] +fn test_factorial() { + println!("factorial({}) = {}", 6, factorial(6)); // This line is here so we can see it in --nocapture + assert_eq!(factorial(6), 720); +} diff --git a/tests/cli/config.rs b/tests/cli/config.rs new file mode 100644 index 00000000..f20ee6bf --- /dev/null +++ b/tests/cli/config.rs @@ -0,0 +1,174 @@ +// Copyright 2022 Martin Pool. + +//! Test handling of `mutants.toml` configuration. + +use std::fs::{create_dir, write}; + +use predicates::prelude::*; +use tempfile::TempDir; + +use super::{copy_of_testdata, run_assert_cmd}; + +fn write_config_file(tempdir: &TempDir, config: &str) { + let path = tempdir.path(); + // This will error if it exists, which today it never will, + // but perhaps later we should ignore that. + create_dir(path.join(".cargo")).unwrap(); + write(path.join(".cargo/mutants.toml"), config.as_bytes()).unwrap(); +} + +#[test] +fn invalid_toml_rejected() { + let testdata = copy_of_testdata("well_tested"); + write_config_file( + &testdata, + r#"what even is this? + "#, + ); + run_assert_cmd() + .args(["mutants", "--list-files", "-d"]) + .arg(testdata.path()) + .assert() + .failure() + .stderr(predicates::str::contains("Error: parse toml from ")); +} + +#[test] +fn invalid_field_rejected() { + let testdata = copy_of_testdata("well_tested"); + write_config_file( + &testdata, + r#"wobble = false + "#, + ); + run_assert_cmd() + .args(["mutants", "--list-files", "-d"]) + .arg(testdata.path()) + .assert() + .failure() + .stderr( + predicates::str::contains("Error: parse toml from ") + .and(predicates::str::contains("unknown field `wobble`")), + ); +} + +#[test] +fn list_with_config_file_exclusion() { + let testdata = copy_of_testdata("well_tested"); + write_config_file( + &testdata, + r#"exclude_globs = ["src/*_mod.rs"] + "#, + ); + run_assert_cmd() + .args(["mutants", "--list-files", "-d"]) + .arg(testdata.path()) + .assert() + .success() + .stdout(predicates::str::contains("_mod.rs").not()); + run_assert_cmd() + .args(["mutants", "--list", "-d"]) + .arg(testdata.path()) + .assert() + .success() + .stdout(predicates::str::contains("_mod.rs").not()); +} + +#[test] +fn list_with_config_file_inclusion() { + let testdata = copy_of_testdata("well_tested"); + write_config_file( + &testdata, + r#"examine_globs = ["src/*_mod.rs"] + "#, + ); + run_assert_cmd() + .args(["mutants", "--list-files", "-d"]) + .arg(testdata.path()) + .assert() + .success() + .stdout(predicates::str::diff( + "src/inside_mod.rs +src/item_mod.rs\n", + )); + run_assert_cmd() + .args(["mutants", "--list", "-d"]) + .arg(testdata.path()) + .assert() + .success() + .stdout(predicates::str::contains("simple_fns.rs").not()); +} + +#[test] +fn list_with_config_file_regexps() { + let testdata = copy_of_testdata("well_tested"); + write_config_file( + &testdata, + r#" + # comments are ok + examine_re = ["divisible"] + exclude_re = ["-> bool with true"] + "#, + ); + run_assert_cmd() + .args(["mutants", "--list", "-d"]) + .arg(testdata.path()) + .assert() + .success() + .stdout(predicates::str::diff( + "src/simple_fns.rs:17: replace divisible_by_three -> bool with false\n", + )); +} + +#[test] +fn tree_fails_without_needed_feature() { + // The point of this tree is to check that Cargo features can be turned on, + // but let's make sure it does fail as intended if they're not. + let testdata = copy_of_testdata("fails_without_feature"); + run_assert_cmd() + .args(["mutants", "-d"]) + .arg(testdata.path()) + .assert() + .failure() + .stdout(predicates::str::contains( + "test failed in an unmutated tree", + )); +} + +#[test] +fn additional_cargo_args() { + // The point of this tree is to check that Cargo features can be turned on, + // but let's make sure it does fail as intended if they're not. + let testdata = copy_of_testdata("fails_without_feature"); + write_config_file( + &testdata, + r#" + additional_cargo_args = ["--features", "needed"] + "#, + ); + run_assert_cmd() + .args(["mutants", "-d"]) + .arg(testdata.path()) + .assert() + .success() + .stdout(predicates::str::contains("1 caught")); +} + +#[test] +fn additional_cargo_test_args() { + // The point of this tree is to check that Cargo features can be turned on, + // but let's make sure it does fail as intended if they're not. + let testdata = copy_of_testdata("fails_without_feature"); + write_config_file( + &testdata, + r#" + additional_cargo_test_args = ["--all-features", ] + "#, + ); + run_assert_cmd() + .args(["mutants", "-d"]) + .arg(testdata.path()) + .assert() + .success() + .stdout(predicates::str::contains("1 caught")); +} diff --git a/tests/cli/main.rs b/tests/cli/main.rs index 919d9cbb..3fba6cd8 100644 --- a/tests/cli/main.rs +++ b/tests/cli/main.rs @@ -23,6 +23,7 @@ use regex::Regex; use subprocess::{Popen, PopenConfig, Redirection}; use tempfile::{tempdir, TempDir}; +mod config; mod jobs; /// A timeout for a `cargo mutants` invocation from the test suite. Needs to be @@ -563,7 +564,7 @@ fn small_well_tested_tree_is_clean() { "\ *** mutation diff: --- src/lib.rs -+++ replace factorial with Default::default() ++++ replace factorial -> u32 with Default::default() @@ -1,17 +1,13 @@" )); assert!(log_content.contains( diff --git a/tests/cli/snapshots/cli__integration_test_source_is_not_mutated__caught.txt.snap b/tests/cli/snapshots/cli__integration_test_source_is_not_mutated__caught.txt.snap index d3e03c55..312af27f 100644 --- a/tests/cli/snapshots/cli__integration_test_source_is_not_mutated__caught.txt.snap +++ b/tests/cli/snapshots/cli__integration_test_source_is_not_mutated__caught.txt.snap @@ -1,6 +1,6 @@ --- -source: tests/cli.rs +source: tests/cli/main.rs expression: content --- -src/lib.rs:1: replace double with Default::default() +src/lib.rs:1: replace double -> u32 with Default::default() diff --git a/tests/cli/snapshots/cli__list_mutants_in_all_trees_as_json.snap b/tests/cli/snapshots/cli__list_mutants_in_all_trees_as_json.snap index e44cd428..621e309d 100644 --- a/tests/cli/snapshots/cli__list_mutants_in_all_trees_as_json.snap +++ b/tests/cli/snapshots/cli__list_mutants_in_all_trees_as_json.snap @@ -127,6 +127,21 @@ expression: buf ] ``` +## testdata/tree/fails_without_feature + +```json +[ + { + "package": "cargo-mutants-testdata-fails-without-feature", + "file": "src/bin/factorial.rs", + "line": 9, + "function": "factorial", + "return_type": "-> u32", + "replacement": "Default::default()" + } +] +``` + ## testdata/tree/hang_avoided_by_attr ```json diff --git a/tests/cli/snapshots/cli__list_mutants_in_all_trees_as_text.snap b/tests/cli/snapshots/cli__list_mutants_in_all_trees_as_text.snap index 57b24225..ff9b45da 100644 --- a/tests/cli/snapshots/cli__list_mutants_in_all_trees_as_text.snap +++ b/tests/cli/snapshots/cli__list_mutants_in_all_trees_as_text.snap @@ -55,6 +55,12 @@ src/bin/factorial.rs:1: replace main with () src/bin/factorial.rs:7: replace factorial -> u32 with Default::default() ``` +## testdata/tree/fails_without_feature + +``` +src/bin/factorial.rs:9: replace factorial -> u32 with Default::default() +``` + ## testdata/tree/hang_avoided_by_attr ``` diff --git a/tests/cli/snapshots/cli__list_mutants_with_diffs_in_factorial.snap b/tests/cli/snapshots/cli__list_mutants_with_diffs_in_factorial.snap index 37e69a50..a3059d74 100644 --- a/tests/cli/snapshots/cli__list_mutants_with_diffs_in_factorial.snap +++ b/tests/cli/snapshots/cli__list_mutants_with_diffs_in_factorial.snap @@ -1,5 +1,5 @@ --- -source: tests/cli.rs +source: tests/cli/main.rs expression: "String::from_utf8_lossy(&output.stdout)" --- src/bin/factorial.rs:1: replace main with () @@ -22,7 +22,7 @@ src/bin/factorial.rs:1: replace main with () src/bin/factorial.rs:7: replace factorial -> u32 with Default::default() --- src/bin/factorial.rs -+++ replace factorial with Default::default() ++++ replace factorial -> u32 with Default::default() @@ -1,19 +1,15 @@ fn main() { for i in 1..=6 { diff --git a/tests/cli/snapshots/cli__uncaught_mutant_in_factorial__caught.txt.snap b/tests/cli/snapshots/cli__uncaught_mutant_in_factorial__caught.txt.snap index f8494b7e..35ab83a4 100644 --- a/tests/cli/snapshots/cli__uncaught_mutant_in_factorial__caught.txt.snap +++ b/tests/cli/snapshots/cli__uncaught_mutant_in_factorial__caught.txt.snap @@ -1,5 +1,6 @@ --- -source: tests/cli.rs +source: tests/cli/main.rs expression: content --- -src/bin/factorial.rs:7: replace factorial with Default::default() +src/bin/factorial.rs:7: replace factorial -> u32 with Default::default() + diff --git a/tests/cli/snapshots/cli__unviable_mutation_of_struct_with_no_default__unviable.txt.snap b/tests/cli/snapshots/cli__unviable_mutation_of_struct_with_no_default__unviable.txt.snap index 5e104d4c..31f04158 100644 --- a/tests/cli/snapshots/cli__unviable_mutation_of_struct_with_no_default__unviable.txt.snap +++ b/tests/cli/snapshots/cli__unviable_mutation_of_struct_with_no_default__unviable.txt.snap @@ -1,6 +1,6 @@ --- -source: tests/cli.rs +source: tests/cli/main.rs expression: content --- -src/lib.rs:11: replace make_an_s with Default::default() +src/lib.rs:11: replace make_an_s -> S with Default::default() diff --git a/tests/cli/snapshots/cli__well_tested_tree_finds_no_problems__caught.txt.snap b/tests/cli/snapshots/cli__well_tested_tree_finds_no_problems__caught.txt.snap index ba9997f0..a432d363 100644 --- a/tests/cli/snapshots/cli__well_tested_tree_finds_no_problems__caught.txt.snap +++ b/tests/cli/snapshots/cli__well_tested_tree_finds_no_problems__caught.txt.snap @@ -1,20 +1,20 @@ --- -source: tests/cli.rs +source: tests/cli/main.rs expression: content --- -src/inside_mod.rs:3: replace outer::inner::name with Default::default() +src/inside_mod.rs:3: replace outer::inner::name -> &'static str with Default::default() src/methods.rs:16: replace Foo::double with () -src/methods.rs:22: replace ::fmt with Ok(Default::default()) -src/methods.rs:28: replace ::fmt with Ok(Default::default()) -src/nested_function.rs:1: replace has_nested with Default::default() -src/nested_function.rs:2: replace has_nested::inner with Default::default() -src/result.rs:5: replace simple_result with Ok(Default::default()) -src/result.rs:9: replace error_if_negative with Ok(Default::default()) +src/methods.rs:22: replace ::fmt -> fmt::Result with Ok(Default::default()) +src/methods.rs:28: replace ::fmt -> fmt::Result with Ok(Default::default()) +src/nested_function.rs:1: replace has_nested -> u32 with Default::default() +src/nested_function.rs:2: replace has_nested::inner -> u32 with Default::default() +src/result.rs:5: replace simple_result -> Result<&'static str, ()> with Ok(Default::default()) +src/result.rs:9: replace error_if_negative -> Result<(), ()> with Ok(Default::default()) src/simple_fns.rs:7: replace returns_unit with () -src/simple_fns.rs:12: replace returns_42u32 with Default::default() -src/simple_fns.rs:17: replace divisible_by_three with true -src/simple_fns.rs:17: replace divisible_by_three with false -src/simple_fns.rs:26: replace double_string with "".into() -src/simple_fns.rs:26: replace double_string with "xyzzy".into() -src/struct_with_lifetime.rs:14: replace Lex<'buf>::buf_len with Default::default() +src/simple_fns.rs:12: replace returns_42u32 -> u32 with Default::default() +src/simple_fns.rs:17: replace divisible_by_three -> bool with true +src/simple_fns.rs:17: replace divisible_by_three -> bool with false +src/simple_fns.rs:26: replace double_string -> String with "".into() +src/simple_fns.rs:26: replace double_string -> String with "xyzzy".into() +src/struct_with_lifetime.rs:14: replace Lex<'buf>::buf_len -> usize with Default::default()