v0.38.0

This public release has been a long-time coming, with a huge number significant changes and improvements being added in the last year!

New Features and Major Changes

• Combine with spfs by @rydrman in #440
  The https://github.com/imageworks/spfs repository has been archived and the code migrated into this one for easier build/test and release processes.

• Fuse Backend (First Pass) by @rydrman in #657
  Spfs and spk can now be run using a FUSE filesystem backend to avoid the use of file localization and OverlayFS. OverlayFS can be combined with FUSE in order to still allow runtime editing and package creation. Use the spfs config setting mount.backend with FuseOnly or FuseWithOverlayFS to enable.

• Spec File Versioning via Traits by @rydrman in #327
  To better support breaking changes in the spec file format, we introduce an api version that should be specified in all spec files moving forward.

• Better Source Package Handing by @rydrman in #428

• Further enhancements to spfs clean by @jrray in #454

• Better Yaml Errors by @rydrman in #499

• Add filter mode for components file filtering by @jrray in #518

• Support pip converting packages that use a zip file by @jrray in #556

• Add filter concept for when collecting files after build by @jrray in #559

• Adjust warning/error behavior of spk rm by @jrray in #585

• Add 'package/version@build' support to more commands by @dcookspi in #574

• Add spfs runtime prune by @jrray in #610

• Add a simple spk lint command by @rydrman in #600
  This will be expanded in the future, but aims to allow local and CI validation as well as provide a place to encode best-practices and warnings for possible gotchas.

• Adding Impossible request checking to the solver by @dcookspi in #506
  Improves the time-to-fail in some common resolves where the requests can be identified as impossible. Spk now runs a solver with and without these checks at the same time and let's them race. In cases where this happens the resolve fails much sooner, but it can also slow down legitimately complex resolves that are possible.

• Refactor check command to allow for repairs of subtrees by @paxsonsa in #602

• Buffered, Compressed Uploads and Downloads by @rydrman in #640

• Use file descriptors in render process, add progress bars by @rydrman in #641

• Spfs ls-tags --recursive by @rydrman in #644

• Add a debug cli command by @jrray in #524
  This command can be used to install source packages for existing packages in an environment when debugging issues with the software.

• Adds search space size calculations with the --show-search-size option by @dcookspi in #620

• Add an Spk Config File by @rydrman in #661

• Adds 'statsd' feature to send statsd metrics from spk by @dcookspi in #565

• Package Templating (va Liquid) by @rydrman in #513

• Extend "@stage" syntax to allow variant to be specified by @jrray in #493

• Add reporting of render statistics by @jrray in #668

• Don't use expect when launching tcsh/csh by @rydrman in #700

• Allow for custom shell messages by @rydrman in #706

• Add --check flag to recover from missing objects by @rydrman in #725

• Add --pid flag to spfs-join by @jrray in #734

• Change --local-repo short flag to -L by @jrray in #751

• Spfs ls upgrade by @nicholy in #689

• Spk du cmd by @nicholy in #774

• Adds 'spfs info /spfs/file/path' support by @dcookspi in #687

• Support conditional install requirements by @jrray in #849

• Make 'spk env package.yaml@build' start an editable /spfs environment by @dcookspi in #842

• Fix illegal env vars by @krishaiquin in #793

Fixes and Minor Changes

• Fix spk test package@build failing by @jrray in #435
• Add "variant-count" alias for num-variants command by @jrray in #438
• Add --tmpdir flag to spfs-enter by @jrray in #448
• Fix failed to read recipe by @rydrman in #453
• Make spk rpm obsolete & replace spfs one by @rydrman in #455
• Change clean_render to clean all users by @jrray in #449
• Sentry context and message tweaks by @dcookspi in #446
• Fix renders having wrong ownership / mode by @jrray in #471
• Improve error messages for IO-related errors by @jrray in #461
• Bugfix for /src package requests with empty component lists by @dcookspi in #474
• Add a way to track the owner of an embedded package by @jrray in #415
• Use --no-runtime flag to avoid recursive runtime creation by @jrray in #483
• Change PkgOpt::validate to use intersects by @jrray in #498
• Add defaults for all metadata fields by @rydrman in #482
• Adding a display limit and additional collation for frequent errors in end of solve report by @dcookspi in #501
• Import Configuration and Efficiency by @rydrman in #487
• Fill out incomplete error message by @rydrman in #521
• Fix bug where static values conflicted with defaults by @rydrman in #522
• Change test additional requirements request order by @jrray in #523
• Experimental solver progress UI by @jrray in #374
• Bugfixes for empty package layers and components in spk bake output by @dcookspi in #510
• Add cap_dac_override to spfs-clean by @jrray in #542
• Fix CompatRange::is_satisfied_by wrong Compatible case by @jrray in #550
• Update spk's spfs repository caches to use DashMap by @dcookspi in #535
• Add retry for copy renders in spfs by @rydrman in #569
• Remove redundant solve in binary build by @rydrman in #557
• Handle "object is not a blob" in spfs clean by @jrray in #578
• Set sync policy when syncing tag in spk-launcher by @jrray in #554
• Improve performance of flattening layers by @jrray in #579
• Fix bug parsing envop with separator field by @rydrman in #555
• Allow components to use undeclared default ones by @rydrman in #558
• Add post rule to rpm to mkdir /spfs by @jrray in #584
• Use daemon when launching spfs-monitor by @jrray in #588
• Fix spfs-clean giving up too early by @jrray in #582
• Track flattened layers in runtime by @jrray in #580
• Stop trying to delete the tags or spfs root directory when deleting tags by @jrray in #597
• search bug-fixes for deleted and deprecated packages by @dcookspi in #572
• Separate Ident Types for Different Specificities by @rydrman in #530
• Add mount namespace to runtime data by @jrray in #593
• Report out of space errors with a custom message by @jrray in #612
• Add contextual info to StorageReadError & StorageWriteError by @jrray in #606
• Pass --python-version when pip converting packages by @jrray in #619
• Bugfix for missing build info for "(requested by..." etc lines in solver output by @dcookspi in #625
• Add logo to website and README by @rydrman in #627
• Use exact version matching for @source by @rydrman in #633
• Simplify Solution Type by @rydrman in #599
• Embedded package RequestedBy improvements by @dcookspi in #634
• Fix treatment of embedded stubs in various commands by @jrray in #549
• Add a short flag to info command, disable by default by @rydrman in #647
• Handle "Failed to render" when out of disk space by @jrray in #622
• Stop treating AlreadyExists from rename as success by @jrray in #639
• Run multiple solvers at once, 2 by default by @dcookspi in #568
• Warn on masked files by @jrray in #525
• Bugfix for prerelease interaction with initial request impossible checks by @dcookspi in #654
• Gracefully(ish) handle hard link limits by @rydrman in #648
• Add retry loop back in spfs-monitor when fetching namespace by @jrray in #611
• Bugfix for initial requests impossible checks task ordering issue by @dcookspi in #656
• Alternate implementation for payload repair by @jrray in #626
• Improve check and fix by @rydrman in #642
• Adds repo name and highest versions to solution output by @dcookspi in #616
• Save disk space by hard linking render proxy files when possible by @jrray in #621
• Use OnceCell for spfs CONFIG by @jrray in #664
• Remove BuildIterator::recipe by @jrray in #669
• Update file test to be functional in sh/dash/bash by @rydrman in #672
• Improve clean via Cleaner by @rydrman in #646
• Fix old runners not existing by @rydrman in #680
• Add limits to check process by @rydrman in #675
• Fix zombie spfs-monitor caused by lost pid events by @jrray in #591
• Variant Trait by @rydrman in #635
• Commit Configuration via Committer Type by @rydrman in #658
• Improve messages for MustNotAlterExistingFiles build errors by @dcookspi in #561
• Add repo name and components to packages output by spk bake by @dcookspi in #681
• Adds error handling to initial request checks to avoid panics by @dcookspi in #685
• Bugfix adds missing send_solver_start_metrics() call by @dcookspi in #686
• Replace parse_ident call with VersionIdent::new by @jrray in #693
• spk-convert-pip: handle another dependency requirement syntax by @jrray in #694
• Remove spfs-enter from process tree by @rydrman in #695
• has_* improvements by @rydrman in #671
• Update logging configuration for sentry & syslog by @jrray in #604
• Bugfix for invalid package handling in search space size calcs by @dcookspi in #697
• Restore env filter when sentry disabled by @rydrman in #699
• Require specific build when testing in install phase by @jrray in #583
• Speed up spk info inside a large runtime by 203x by @jrray in #679
• Use debug print for anyhow errors in spfs by @rydrman in #704
• Fix missing arg to configure_sentry by @jrray in #707
• Use exec to replace shell during spfs run by @rydrman in #710
• Faster package validation by @jrray in #713
• Improve error message when spfs-render fails (simple) by @jrray in #726
• Also repair blobs in PayloadFallback by @jrray in #717
• Update to multiple solver selection and finished solver messages by @dcookspi in #696
• Stops 'spfs run' syncing tags by @dcookspi in #714
• spk test in install phase not looking for correct build digest by @jrray in #729
• Respect components on build options by @jrray in #723
• Make spfs-join more reliable by @jrray in #733
• Adds sync_time and render_time support to spfs enter by @dcookspi in #682
• Fix another case of spk test not picking the correct build hash by @jrray in #730
• Fixes for misleading help messages for 'spfs runtime' prune and remove subcommands by @dcookspi in #735
• Adds a check for an --env-name value and uses it to name spfs runtimes by @dcookspi in #736
• Add rules for doing release from release branches by @jrray in #738
• Rename --name and --env-name args to --runtime-name in spfs/spk by @dcookspi in #741
• Fix infinite loop attempting to repair missing payload by @jrray in #745
• Add wheel to spk-convert-pip requirements by @jrray in #752
• Overlayfs basic supported features check and default inode index by @rydrman in #502
• Fix remove_package not removing build spec by @jrray in #746
• Runtime Edit Flag Fixes by @rydrman in #757
• Fix spfs-monitor hanging on sentry close by @jrray in #773
• Do not print newline when shell message is empty by @rydrman in #771
• Add explicit permissions for created fs objects by @rydrman in #772
• Add more output control and file to spfs-monitor by @dcookspi in #769
• Limit packages through spk debug by @nicholy in #780
• Authors metadata in workspace by @nicholy in #784
• Fix spk export failing if builds are missing by @jrray in #800
• Also fallback to copying on too many links for proxy files by @rydrman in #795
• Remove duplicate reporting of entries in render report by @rydrman in #801
• Clarify what the letters mean in spfs and spk by @rydrman in #786
• Add timestamp to CI log lines by @nicholy in #798
• Fixed issue where spk publish will overwrite versions without --force by @nicholy in #791
• Improve spk experience using fuse backend by @jrray in #810
• Allow 'spfs runtime remove|rm' to remove multiple runtimes by @dcookspi in #804
• Remove cargo build step when linting packages by @jrray in #818
• Implement "Strong" inheritance by @jrray in #811
• Change statsd configuration to be optional by @jrray in #806
• Implemented missing features for install.environment by @nicholy in #782
• Support "extras" in spk-convert-pip by @jrray in #767
• Bugfix: spk bake outputs layers for requested components not all components by @dcookspi in #826
• Spfs log output to show local time instead of UTC time. by @nicholy in #831
• Windows SPFS Build by @rydrman in #829
• Fix spk rm taking 5 seconds per package deleted by @jrray in #834
• Output size of each layer when running spfs info by @nicholy in #838
• Fix spk build build digest calculation by @jrray in #765
• Move package requests matching "platform" to the head of the request list by @jrray in #824
• Fix spfs clean not working on some hosts by @jrray in #845
• Start using spk config file for various things by @jrray in #825
• Sort packages inside active runtime when using spk info by @nicholy in #836
• Fix lost deletion error in remove_package_from_storage by @jrray in #857
• Add configurable tokio thread limits to spfs by @rydrman in #854
• Add env var to suppress overlayfs param warning by @jrray in #868

Package Specs

• packages: use fromBuildEnv: Binary and minor enhancements by @davvid in #437
• Fix packages makefile by @rydrman in #476
• Update makefile for linting and testing specific crates by @rydrman in #486
• packages: add ninja by @davvid in #536
• packages: add opentimelineio by @davvid in #537
• Fix broken Makefile check for src package by @rydrman in #651
• Fix invalid version numbers in m4 package by @rydrman in #728

New Contributors

• @jdriscoll1 made their first contribution in #494
• @paxsonsa made their first contribution in #602
• @nicholy made their first contribution in #689
• @krishaiquin made their first contribution in #793

Full Changelog: v0.36.0...v0.38.0