Skip to content

Latest commit

 

History

History
186 lines (127 loc) · 6.89 KB

README.md

File metadata and controls

186 lines (127 loc) · 6.89 KB

SPK

Package Manager for SPFS.

Motivation

SpFS provides a powerful set of tools for capturing and isolating process filesystems at runtime, but not a lot of great workflows for managing and organizing the environments and layers. SPK is the solution to this problem, adding the concept of software packages and the process of environment and dependency resolution for a better workflow.

Usage

See the main docs for details on using spk, starting with the index.

License

SPK/SPFS/spawn are Copyright (c) 2021 Sony Pictures Imageworks, et al. All Rights Reserved.

SPK/SPFS/spawn are distributed using the Apache-2.0 license.

Structure of the spk project

spfs is the per-process layered file system.

spk is the software packaging system built on top of SPFS.

spawn is the application launcher for spk packages.

These are spread over three code bases at the moment, but will probably be merged into a single project, spk. Please refer to spk for almost all information about staging the open source project, that's where the developer documentation and communication will live, including Contributing to SPK.

Contributing

Please read Contributing to SPK.

Development plan

Please read SPK open source development plan.

Development

SPK is mostly written in python, with a Rust extension that integrates with the spfs API.

For details on architecture and design of the codebase, see the developer docs.

Python dependencies are tracked with Pipenv, which will need to be installed. You will also need access to the rust toolchain, which can be installed with rustup.

Once you have access to the pipenv command, jump into a development environment using:

pipenv sync --dev
pipenv shell

The easiest way to work with spk is to install a local development version into the pipenv virtaul environment. Once in a pipenv shell, this can be achieved by running the commands below.

# install the local sources into the virtualenv
python setup.py develop
which spk # now points to local dev version
spk --help

NOTE In order to run spk you will need spfs to already be installed on the local system. The easiest way to do this is to download the rpm file attached to the latest release in the spfs repo.

RPM Package

The spk codebase is setup to produce a centos7-compatible rpm package by building spfs in a docker container. To create the rpm package, you will need docker installed. These packages are also built and made available in this repository's CI.

In order to properly build the rpm, you will need to provide your github username and an access token so that the container can pull the spfs sources to build against. The Makefile is setup to prompt you for and fill in these values automatically. If you don't wish to fill these in each time, you can also set the SPFS_PULL_USERNAME and SPFS_PULL_PASSWORD environment variables before calling make.

# build the rpm package via docker and copy into ./dist/rpm
make rpm

Testing

Spfs has a number of unit and integration tests as well as testable examples that can all be executed with pytest. The tests themselves need to be executed under an spfs runtime in order to properly execute.

spfs run - -- pytest

Additionally, there are some rust unit tests that can be executed using cargo.

make cargo-test

From this shell, you can run the local build of the spk command as well as all tests with pytest. NOTE: running the local development version of spk, and running the unit tests will require that spfs is installed on the local machine. The pytest test suite must also be run from within an spfs environment in order to work properly.

# run the unit test suite
make test

Bootstrapping

In a new environment, it can be helpful to build all of the core packages whose recipes ship with SPK. A script has been provided which runs through all of the builds for these packages in the right order.

# bootstrap and build all core packages (takes a long time)
make packages
# build only enough to bootstrap a compiler and linker
make packages.bootstrap
# build the python2 package (will require at least packages.bootstrap)
make packages.python2

Some of these package specs have not yet been used or tested fully or ironed out all the way so please communicate any issues as you run into them!

Using Docker

Currently, this process can only be run on an rpm-based system, as it relies on some rpm packages being installed on the host in order to bootstrap the build process. If you are not running on an rpm-based system, you can run the process in a container instead:

# build boostrap packages in a docker image
# (can also build any other packages.* rule, though the container startup is heavy)
make packages.docker.python2
# build all core packages
make packages.docker
# import the created packages to the local spk environment
make packages.import

Conversion Packages

Spk has logic to automatically convert pip packages to spk packages for easy python environment creation. This logic lives and runs inside of it's own spk package/environment. If you have python3 already installed, you can generate this package locally like so:

make converters

Once built, these packages will need to be published in order to use them from the spk convert command.

make converters
spk publish spk-convert-pip/1.0.0
spk convert pip --help

Other Notes

  • The make packages.python2 and packages.python3 targets can be used to boostrap just enough to be able to build python for spk. The python recipes will build multiple python versions for each gcc48 and 63 as well as for the different python abi's
  • The make packages.gnu target can be used to bootstrap just enough to get "native" spk packages for gcc48 and gcc63

Of course, the packages themselves can also be build with the spk build <spec_file> command directly, though you may find that some required build dependencies need to be generated with the make packages.bootstrap.full command first.

The following RPM packages must be installed in order to create the bootstrap packages.

sudo yum install -y \
    autoconf \
    autoconf-archive \
    autogen \
    automake \
    binutils \
    bison \
    coreutils \
    flex \
    gcc \
    gettext \
    glibc \
    grep \
    help2man \
    libtool \
    m4 \
    make \
    perl \
    sed \
    texinfo \
    zip \
    zlib