Skip to content

Commit

Permalink
Respect gitignore (#169)
Browse files Browse the repository at this point in the history
  • Loading branch information
sourcefrog authored Nov 25, 2023
2 parents ff14f55 + 5c8ff7e commit 46e36df
Show file tree
Hide file tree
Showing 21 changed files with 308 additions and 94 deletions.
18 changes: 18 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ ctrlc = { version = "3.2.1", features = ["termination"] }
fastrand = "2"
fs2 = "0.4"
globset = "0.4.8"
ignore = "0.4.20"
indoc = "2.0.0"
itertools = "0.11"
mutants = "0.0.3"
Expand All @@ -61,9 +62,6 @@ tracing-appender = "0.2"
tracing-subscriber = "0.3"
whoami = "1.2"

[dependencies.cp_r]
version = "0.5.1"

[dependencies.nutmeg]
version = "0.1.4"
# git = "https://github.com/sourcefrog/nutmeg.git"
Expand All @@ -83,6 +81,7 @@ features = ["full", "extra-traits", "visit"]

[dev-dependencies]
assert_cmd = "2.0"
cp_r = "0.5.1"
insta = "1.12"
lazy_static = "1.4"
predicates = "3"
Expand Down Expand Up @@ -125,6 +124,7 @@ exclude = [
"testdata/small_well_tested",
"testdata/strict_warnings",
"testdata/struct_with_no_default",
"testdata/symlink",
"testdata/unapply",
"testdata/unsafe",
"testdata/well_tested",
Expand Down
8 changes: 8 additions & 0 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,13 @@ See also [CONTRIBUTING.md](CONTRIBUTING.md) for more advice on style, approach,

Actually running subprocesses is delegated to `process.rs`, so that we can later potentially run different build tools to Cargo.

`build_dir.rs` -- Manage temporary build directories.

`console.rs` -- colored output to the console including drawing progress bars.
The interface to the `console` and `indicatif` crates is localized here.

`copy_tree.rs` -- Copy a source file tree into a build dir, with gitignore and other exclusions.

`interrupt.rs` -- Handle Ctrl-C signals by setting a global atomic flag, which
is checked during long-running operations.

Expand Down Expand Up @@ -99,6 +103,10 @@ scratch directory.

Currently, the whole workspace tree is copied. In future, possibly only the package to be mutated could be copied: this would require changes to the code that fixes up dependencies.

Copies by default respect gitignore, but this can be turned off.

Each parallel build dir is copied from the original source so that it sees any gitignore files in parent directories.

(This current approach assumes that all the packages are under the workspace directory, which is common but not actually required.)

## Handling timeouts
Expand Down
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

- Changed: If `--file` or `--exclude` are set on the command line, then they replace the corresponding config file options. Similarly, if `--re` is given then the `examine_re` config key is ignored, and if `--exclude-re` is given then `exclude_regex` is ignored. (Previously the values were combined.) This makes it easier to use the command line to test files or mutants that are normally not tested.

- Improved: By default, files matching gitignore patterns (including in parent directories, per-user configuration, and `info/exclude`) are excluded from copying to temporary build directories. This should improve performance in some large trees with many files that are not part of the build. This behavior can be turned off with `--gitignore=false`.

- Improved: Run `cargo metadata` with `--no-deps`, so that it doesn't download and compute dependency information, which can save time in some situations.

- Added: Alternative aliases for command line options, so you don't need to remember if it's "regex" or "re": `--regex`, `--examine-re`, `--examine-regex` (all for names to include) and `--exclude-regex`.
Expand Down
1 change: 1 addition & 0 deletions book/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
- [Listing and previewing mutations](list.md)
- [Workspaces and packages](workspaces.md)
- [Passing options to Cargo](cargo-args.md)
- [Build directories](build-dirs.md)
- [Generating mutants](mutants.md)
- [Error values](error-values.md)
- [Improving performance](performance.md)
Expand Down
20 changes: 20 additions & 0 deletions book/src/build-dirs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Build directories

cargo-mutants builds mutated code in a temporary directory, containing a copy of your source tree with each mutant successively applied. With `--jobs`, multiple build directories are used in parallel.

## Build-in ignores

Files or directories matching these patterns are not copied:

.git
.hg
.bzr
.svn
_darcs
.pijul

## gitignore

From 23.11.2, by default, cargo-mutants will not copy files that are excluded by gitignore patterns, to make copying faster in large trees.

This behavior can be turned off with `--gitignore=false`.
88 changes: 5 additions & 83 deletions src/build_dir.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,18 @@
use std::convert::TryInto;
use std::fmt;

use anyhow::Context;
use camino::{Utf8Path, Utf8PathBuf};
use tempfile::TempDir;
use tracing::{debug, error, info, trace};
use tracing::info;

use crate::copy_tree::copy_tree;
use crate::manifest::fix_cargo_config;
use crate::*;

/// Filenames excluded from being copied with the source.
const SOURCE_EXCLUDE: &[&str] = &[
".git",
".hg",
".bzr",
".svn",
"_darcs",
".pijul",
"mutants.out",
"mutants.out.old",
"target",
];

/// A temporary directory initialized with a copy of the source, where mutations can be tested.
pub struct BuildDir {
/// The path of the root of the temporary directory.
path: Utf8PathBuf,
/// A prefix for tempdir names, based on the name of the source directory.
name_base: String,
/// Holds a reference to the temporary directory, so that it will be deleted when this
/// object is dropped.
#[allow(dead_code)]
Expand All @@ -48,12 +33,11 @@ impl BuildDir {
///
/// [SOURCE_EXCLUDE] is excluded.
pub fn new(source: &Utf8Path, options: &Options, console: &Console) -> Result<BuildDir> {
let name_base = format!("cargo-mutants-{}-", source.file_name().unwrap_or(""));
let name_base = format!("cargo-mutants-{}-", source.file_name().unwrap_or("unnamed"));
let source_abs = source
.canonicalize_utf8()
.expect("canonicalize source path");
// TODO: Only exclude `target` in directories containing Cargo.toml?
let temp_dir = copy_tree(source, &name_base, SOURCE_EXCLUDE, console)?;
let temp_dir = copy_tree(source, &name_base, options.gitignore, console)?;
let path: Utf8PathBuf = temp_dir.path().to_owned().try_into().unwrap();
fix_manifest(&path.join("Cargo.toml"), &source_abs)?;
fix_cargo_config(&path, &source_abs)?;
Expand All @@ -64,28 +48,13 @@ impl BuildDir {
} else {
TempDirStrategy::Collect(temp_dir)
};
let build_dir = BuildDir {
strategy,
name_base,
path,
};
let build_dir = BuildDir { strategy, path };
Ok(build_dir)
}

pub fn path(&self) -> &Utf8Path {
self.path.as_path()
}

/// Make a copy of this build dir, including its target directory.
pub fn copy(&self, console: &Console) -> Result<BuildDir> {
let temp_dir = copy_tree(&self.path, &self.name_base, &[], console)?;

Ok(BuildDir {
path: temp_dir.path().to_owned().try_into().unwrap(),
strategy: TempDirStrategy::Collect(temp_dir),
name_base: self.name_base.clone(),
})
}
}

impl fmt::Debug for BuildDir {
Expand All @@ -96,53 +65,6 @@ impl fmt::Debug for BuildDir {
}
}

fn copy_tree(
from_path: &Utf8Path,
name_base: &str,
exclude: &[&str],
console: &Console,
) -> Result<TempDir> {
let temp_dir = tempfile::Builder::new()
.prefix(name_base)
.suffix(".tmp")
.tempdir()
.context("create temp dir")?;
console.start_copy();
let copy_options = cp_r::CopyOptions::new()
.after_entry_copied(|path, _ft, stats| {
console.copy_progress(stats.file_bytes);
check_interrupted().map_err(|_| cp_r::Error::new(cp_r::ErrorKind::Interrupted, path))
})
.filter(|path, _dir_entry| {
let excluded = exclude.iter().any(|ex| path.ends_with(ex));
if excluded {
trace!("Skip {path:?}");
} else {
trace!("Copy {path:?}");
}
Ok(!excluded)
});
match copy_options
.copy_tree(from_path, temp_dir.path())
.context("copy tree")
{
Ok(stats) => {
debug!(files = stats.files, file_bytes = stats.file_bytes,);
}
Err(err) => {
error!(
"error copying {} to {}: {:?}",
&from_path.to_slash_path(),
&temp_dir.path().to_slash_lossy(),
err
);
return Err(err);
}
}
console.finish_copy();
Ok(temp_dir)
}

#[cfg(test)]
mod test {
use regex::Regex;
Expand Down
130 changes: 130 additions & 0 deletions src/copy_tree.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
// Copyright 2023 Martin Pool

//! Copy a source tree, with some exclusions, to a new temporary directory.
use std::convert::TryInto;
use std::fs::FileType;

use anyhow::Context;
use camino::{Utf8Path, Utf8PathBuf};
use ignore::WalkBuilder;
use path_slash::PathExt;
use tempfile::TempDir;
use tracing::{debug, warn};

use crate::check_interrupted;
use crate::Console;
use crate::Result;

/// Filenames excluded from being copied with the source.
static SOURCE_EXCLUDE: &[&str] = &[
".git",
".hg",
".bzr",
".svn",
"_darcs",
".pijul",
"mutants.out",
"mutants.out.old",
];

/// Copy a source tree, with some exclusions, to a new temporary directory.
///
/// If `git` is true, ignore files that are excluded by all the various `.gitignore`
/// files.
///
/// Regardless, anything matching [SOURCE_EXCLUDE] is excluded.
pub fn copy_tree(
from_path: &Utf8Path,
name_base: &str,
gitignore: bool,
console: &Console,
) -> Result<TempDir> {
console.start_copy();
let mut total_bytes = 0;
let mut total_files = 0;
let temp_dir = tempfile::Builder::new()
.prefix(name_base)
.suffix(".tmp")
.tempdir()
.context("create temp dir")?;
for entry in WalkBuilder::new(from_path)
.standard_filters(gitignore)
.hidden(false)
.require_git(false)
.filter_entry(|entry| {
!SOURCE_EXCLUDE.contains(&entry.file_name().to_string_lossy().as_ref())
})
.build()
{
check_interrupted()?;
let entry = entry?;
let relative_path = entry
.path()
.strip_prefix(from_path)
.expect("entry path is in from_path");
let dest_path: Utf8PathBuf = temp_dir
.path()
.join(relative_path)
.try_into()
.context("Convert path to UTF-8")?;
let ft = entry
.file_type()
.with_context(|| format!("Expected file to have a file type: {:?}", entry.path()))?;
if ft.is_file() {
let bytes_copied = std::fs::copy(entry.path(), &dest_path).with_context(|| {
format!(
"Failed to copy {:?} to {dest_path:?}",
entry.path().to_slash_lossy(),
)
})?;
total_bytes += bytes_copied;
total_files += 1;
console.copy_progress(bytes_copied);
} else if ft.is_dir() {
std::fs::create_dir_all(&dest_path)
.with_context(|| format!("Failed to create directory {dest_path:?}"))?;
} else if ft.is_symlink() {
copy_symlink(
ft,
entry
.path()
.try_into()
.context("Convert filename to UTF-8")?,
&dest_path,
)?;
} else {
warn!("Unexpected file type: {:?}", entry.path());
}
}
console.finish_copy();
debug!(?total_bytes, ?total_files, "Copied source tree");
Ok(temp_dir)
}

#[cfg(unix)]
fn copy_symlink(_ft: FileType, src_path: &Utf8Path, dest_path: &Utf8Path) -> Result<()> {
let link_target = std::fs::read_link(src_path)
.with_context(|| format!("Failed to read link {src_path:?}"))?;
std::os::unix::fs::symlink(link_target, dest_path)
.with_context(|| format!("Failed to create symlink {dest_path:?}",))?;
Ok(())
}

#[cfg(windows)]
#[mutants::skip] // Mutant tests run on Linux
fn copy_symlink(ft: FileType, src_path: &Utf8Path, dest_path: &Utf8Path) -> Result<()> {
use std::os::windows::fs::FileTypeExt;
let link_target =
std::fs::read_link(src_path).with_context(|| format!("read link {src_path:?}"))?;
if ft.is_symlink_dir() {
std::os::windows::fs::symlink_dir(link_target, dest_path)
.with_context(|| format!("create symlink {dest_path:?}"))?;
} else if ft.is_symlink_file() {
std::os::windows::fs::symlink_file(link_target, dest_path)
.with_context(|| format!("create symlink {dest_path:?}"))?;
} else {
anyhow::bail!("Unknown symlink type: {:?}", ft);
}
Ok(())
}
Loading

0 comments on commit 46e36df

Please sign in to comment.