Preparing the deployment of DUNE DAQ spack releases to cvmfs

The preparation is done on the following two servers at NP04.

np04-srv-016.cern.ch for the deployment to cvmfs sandbox.
np04-srv-015.cern.ch for the production deployment.

It contains three elements:

Prepare the spack instance for externals
Prepare the spack instance for a frozen release
Prepare the pre-built python virtual environment for a release

The directory tree of the deployment (taking the sandbox deployment as an example) should look like:

[pding@np04-srv-016 sandbox]$ tree -L 3 .
.
├── spack-externals
│   ├── spack-0.17.1
│   │   ├── bin
│   │   ├── CHANGELOG.md
│   │   ├── COPYRIGHT
│   │   ├── etc
│   │   ├── lib
│   │   ├── LICENSE-APACHE
│   │   ├── LICENSE-MIT
│   │   ├── NOTICE
│   │   ├── opt
│   │   ├── pyproject.toml
│   │   ├── pytest.ini
│   │   ├── README.md
│   │   ├── SECURITY.md
│   │   ├── share
│   │   └── var
│   ├── spack-repo-externals
│   │   ├── packages
│   │   └── repo.yaml
│   └── spack-repo-N22-04-01
│       ├── N22-04-01.yaml
│       ├── packages
│       └── repo.yaml
├── spack-nightly
│   └── N22-04-02
│       ├── dbt-build-order.cmake
│       ├── dbt-pyvenv
│       ├── pyvenv_requirements.txt
│       ├── spack-0.17.1
│       └── spack-repo
└── spack-releases
    └── dunedaq-v2.10.1
        ├── dbt-build-order.cmake
        ├── dbt-pyvenv
        ├── pyvenv_requirements.txt
        ├── spack-0.17.1
        └── spack-repo

22 directories, 16 files

🔴 Remove the commands of setting http(s)_proxy from the code block if not running on NP04 nodes. 🔴

💡 Code blocks in this document without the following warnings are directly copyable and can be run in batch mode.

🔴 Note: code blocks below does not run in batch mode, ... 🔴

The preparation is done inside docker containers so that we can bind mount local disk to the `cvmfs` path matching the deployment. To start the docker container, do the following (remove `--rm` option in the docker command if you want to log back into the container later). Because the default `glibc` version on CS8 is not backward compatible with those on SL7, it is important to do the preparation with a SL7 container, so that the end product is compatible on both SL7 and CS8.

# Change the following two envs to different path 
# if you are not the owner of `/scratch/dingpf` directory. :)
export LOCAL_CVMFS_DUNEDAQ_DEV=/scratch/dingpf/docker-scratch/cvmfs_dunedaq-dev
export LOCAL_CVMFS_DUNEDAQ=/scratch/dingpf/docker-scratch/cvmfs_dunedaq
mkdir -p $LOCAL_CVMFS_DUNEDAQ_DEV
mkdir -p $LOCAL_CVMFS_DUNEDAQ

docker run --rm -it --net=host \
-v $LOCAL_CVMFS_DUNEDAQ:/cvmfs/dunedaq.opensciencegrid.org \
-v $LOCAL_CVMFS_DUNEDAQ_DEV:/cvmfs/dunedaq-development.opensciencegrid.org \
dunedaq/sl7-minimal:spack

Making the `externals` spack instance

To ensure a consistent concretization result of the externals-only installation with the full DAQ release, the installation takes several steps:

Generate a nightly release repo using the dunedaq-develop release's YAML file, e.g N22-04-01;
Add both the externals repo and the generated release repo to spack;
Concretize on the release's umbrella package, dunedaq@N22-04-01, install the release;
At this point, all the external packages plus DAQ packages have installed. Next run spack uninstall --dependents daq-cmake@N22-04-01. Since daq-cmake is a dependency of all DAQ packages. Using the --dependents option with spack unintall will remove all DAQ packages. Only external packages will be left in the instance beyond this point.

Setting up the directories

For sandbox:

export SPACK_EXTERNALS=/cvmfs/dunedaq-development.opensciencegrid.org/sandbox/spack-externals
mkdir -p $SPACK_EXTERNALS

For production:

export SPACK_EXTERNALS=/cvmfs/dunedaq.opensciencegrid.org/spack-externals
mkdir -p $SPACK_EXTERNALS

Copy over the `externals` repo

export http_proxy=http://np04-web-proxy.cern.ch:3128
export https_proxy=http://np04-web-proxy.cern.ch:3128
cd $HOME
export DAQ_RELEASE_DIR=$HOME/daq-release
git clone https://github.com/DUNE-DAQ/daq-release $DAQ_RELEASE_DIR
cp -pr $DAQ_RELEASE_DIR/spack-repos/externals $SPACK_EXTERNALS
mv $SPACK_EXTERNALS/externals $SPACK_EXTERNALS/spack-repo-externals

Generate a nightly release repo

To avoid possible issues with repo name conflicts, use a nightly release tag for a older date which is not deployed in cvmfs. In this documentation, we use N22-04-01.

cd $DAQ_RELEASE_DIR

python3 scripts/spack/make-release-repo.py -u \
-i configs/dunedaq-develop/dunedaq-develop.yaml \
-t spack-repos/release-repo-template \
-r N22-04-01 \
-o $SPACK_EXTERNALS

mv $SPACK_EXTERNALS/spack-repo $SPACK_EXTERNALS/spack-repo-N22-04-01

Install a spack instance

🔴 Note: code blocks below does not run in batch mode, it requires modifying a file/files interactively. 🔴

cd $SPACK_EXTERNALS
export http_proxy=http://np04-web-proxy.cern.ch:3128
export https_proxy=http://np04-web-proxy.cern.ch:3128
wget https://github.com/spack/spack/archive/refs/tags/v0.17.1.tar.gz
tar xf v0.17.1.tar.gz
ln -s spack-0.17.1 spack-installation
rm -f v0.17.1.tar.gz

# modify the installation prefix for spack packages
vi $SPACK_EXTERNALS/spack-0.17.1/etc/spack/defaults/config.yaml

Install `externals` repo

1. Add externals and nightly release repo to spack, modofy default `repos.yaml`

🔴 Note: code blocks below does not run in batch mode, it requires modifying a file/files interactively. 🔴

cd $SPACK_EXTERNALS
source spack-0.17.1/share/spack/setup-env.sh
spack repo add spack-repo-externals/
spack repo add spack-repo-N22-04-01/

# modify repos.yaml
# NOTE: make sure built-in repo is at the end of the list
cat ~/.spack/repos.yaml >>$SPACK_ROOT/etc/spack/defaults/repos.yaml
vi $SPACK_ROOT/etc/spack/defaults/repos.yaml

rm ~/.spack/repos.yaml

2. Install `gcc@8.2.0`, modify default `compilers.yaml`

spack compiler find
spack install gcc@8.2.0 +binutils
spack load gcc@8.2.0
spack compiler find
cp /root/.spack/linux/compilers.yaml spack-0.17.1/etc/spack/defaults/linux/
rm ~/.spack/linux/compilers.yaml

3. Install the release umbrella package for the nightly release

spack install dunedaq@N22-04-01%gcc@8.2.0

4. Remove all DAQ packages by uninstall `daq-cmake` and its dependents

spack uninstall --dependents daq-cmake@N22-04-01

Making a spack instance for a single frozen release

Once the externals spack instance installation is completed, exit the docker container, and start it again with the same disk mounts. This is to ensure a clean shell environment in which spack is not set up.

export LOCAL_CVMFS_DUNEDAQ_DEV=/scratch/dingpf/docker-scratch/cvmfs_dunedaq-dev
export LOCAL_CVMFS_DUNEDAQ=/scratch/dingpf/docker-scratch/cvmfs_dunedaq

docker run --rm -it --net=host \
-v $LOCAL_CVMFS_DUNEDAQ:/cvmfs/dunedaq.opensciencegrid.org \
-v $LOCAL_CVMFS_DUNEDAQ_DEV:/cvmfs/dunedaq-development.opensciencegrid.org \
dunedaq/sl7-minimal:spack

The steps in this section is using dunedaq-v2.10.1 release as an example.

🔴 If you are preparing the deployment on a different node, firstly copy over the spack-externals directory prepared from the previous step to $LOCAL_CVMFS_DUNEDAQ/spack-externals. Alternatively, you can start the container with the following:🔴

export LOCAL_CVMFS_DUNEDAQ_DEV=/scratch/dingpf/docker-scratch/cvmfs_dunedaq-dev
export LOCAL_CVMFS_DUNEDAQ=/scratch/dingpf/docker-scratch/cvmfs_dunedaq
mkdir -p $LOCAL_CVMFS_DUNEDAQ/spack-releases

docker run --rm -it --net=host \
-v $LOCAL_CVMFS_DUNEDAQ/spack-releases:/cvmfs/dunedaq.opensciencegrid.org/spack-releases \
-v /cvmfs/dunedaq.opensciencegrid.org/spack-externals:/cvmfs/dunedaq.opensciencegrid.org/spack-externals \
-v $LOCAL_CVMFS_DUNEDAQ_DEV:/cvmfs/dunedaq-development.opensciencegrid.org \
dunedaq/sl7-minimal:spack

Setting up the directories

For sandbox:

export SPACK_RELEASE=/cvmfs/dunedaq-development.opensciencegrid.org/sandbox/spack-releases
export SPACK_EXTERNALS=/cvmfs/dunedaq-development.opensciencegrid.org/sandbox/spack-externals
export DAQ_RELEASE=dunedaq-v2.10.1
mkdir -p ${SPACK_RELEASE}/${DAQ_RELEASE}

For production:

export SPACK_RELEASE=/cvmfs/dunedaq.opensciencegrid.org/spack-releases
export SPACK_EXTERNALS=/cvmfs/dunedaq.opensciencegrid.org/spack-externals
export DAQ_RELEASE=dunedaq-v2.10.1
mkdir -p ${SPACK_RELEASE}/${DAQ_RELEASE}

Install a spack instance

cd ${SPACK_RELEASE}/${DAQ_RELEASE}
export http_proxy=http://np04-web-proxy.cern.ch:3128
export https_proxy=http://np04-web-proxy.cern.ch:3128
wget https://github.com/spack/spack/archive/refs/tags/v0.17.1.tar.gz
tar xf v0.17.1.tar.gz
ln -s spack-0.17.1 spack-installation
rm -f v0.17.1.tar.gz

Generating spack repo for a frozen release

## load python3.8.3 and py-pyyaml from externals instance
## this is needed only in the SL7 container
source $SPACK_EXTERNALS/spack-0.17.1/share/spack/setup-env.sh
spack load py-pyyaml@5.3.1

export DAQ_RELEASE_DIR=$HOME/daq-release

git clone https://github.com/DUNE-DAQ/daq-release $DAQ_RELEASE_DIR

cd $DAQ_RELEASE_DIR
python3 scripts/spack/make-release-repo.py -u \
-i configs/${DAQ_RELEASE}/${DAQ_RELEASE}.yaml \
-t spack-repos/release-repo-template \
-o ${SPACK_RELEASE}/${DAQ_RELEASE}

Modify spack default settings to use the `externals` upstream spack instance

🔴 Note: code blocks below does not run in batch mode, it requires modifying a file/files interactively. 🔴

cd $SPACK_RELEASE/$DAQ_RELEASE
source spack-0.17.1/share/spack/setup-env.sh
spack repo add $SPACK_EXTERNALS/spack-repo-externals
cat ~/.spack/repos.yaml >>$SPACK_ROOT/etc/spack/defaults/repos.yaml

# modify repos.yaml, make sure built-in repo is at the end of the list.
vi $SPACK_ROOT/etc/spack/defaults/repos.yaml
rm -f ~/.spack/repos.yaml

# verify the spack repo has been added
spack repo list

# adding externals spack instance as an upstream

cat <<EOT >> $SPACK_ROOT/etc/spack/defaults/upstreams.yaml
upstreams:
  spack-externals:
    install_tree: ${SPACK_EXTERNALS}/spack-0.17.1/opt/spack
EOT

# run "spack find" to make sure the externals insance is successfully added
spack find

# Adding compilers installed in the externals instance to "compilers.yaml"
# We can do this by simply copying over the "compilers.yaml" in the externlas instance
cp $SPACK_EXTERNALS/spack-0.17.1/etc/spack/defaults/linux/compilers.yaml \
$SPACK_RELEASE/$DAQ_RELEASE/spack-0.17.1/etc/spack/defaults/linux/
# verify the compilers have been added
spack compiler list

# modify the installation prefix for spack packages
# here we simply copy over the same settings as the externals instance
# use "\cp" as in RHEL, "cp" is an alias of "cp -i"; "\cp" will do "unalias".
\cp $SPACK_EXTERNALS/spack-0.17.1/etc/spack/defaults/config.yaml  $SPACK_RELEASE/$DAQ_RELEASE/spack-0.17.1/etc/spack/defaults/config.yaml

Install the release in a spack environment

# spack does not allow env names contain any dots 
export DUNEDAQ_SPACK_ENV=${DAQ_RELEASE//./-}
spack env create $DUNEDAQ_SPACK_ENV
spack env activate $DUNEDAQ_SPACK_ENV -p

# adding the release repo
spack repo add $SPACK_RELEASE/$DAQ_RELEASE/spack-repo

# add the release umbrella package to the spack environment
spack add dunedaq@${DAQ_RELEASE}%gcc@8.2.0

# run spack concretize to ensure the concretizer gets a successfully result before installation
spack concretize

# install the release
spack install

If the steps above were executed successfully, assuming there is no changes to the set of external packages, you will see spack starts to install DAQ packages ONLY without any external dependencies at this moment. All external dependencies should have been found in the externals instance, and spack is smart enough to reuse them.

Making a spack instance for a single nightly release

Similar to making a spack instance for a frozen release, start from a fresh docker container with same disk mounts as those used for making externlas instance.

The steps in the sections below uses N22-04-02 as an example. You can replace it with N$(date +%y-%m-%d).

export LOCAL_CVMFS_DUNEDAQ_DEV=/scratch/dingpf/docker-scratch/cvmfs_dunedaq-dev
export LOCAL_CVMFS_DUNEDAQ=/scratch/dingpf/docker-scratch/cvmfs_dunedaq

docker run --rm -it --net=host \
-v $LOCAL_CVMFS_DUNEDAQ:/cvmfs/dunedaq.opensciencegrid.org \
-v $LOCAL_CVMFS_DUNEDAQ_DEV:/cvmfs/dunedaq-development.opensciencegrid.org \
dunedaq/sl7-minimal:spack

🔴 If you are preparing the deployment on a different node, firstly copy over the spack-externals directory prepared from the previous step to $LOCAL_CVMFS_DUNEDAQ/spack-externals. Alternatively, you can start the container with the following:🔴

export LOCAL_CVMFS_DUNEDAQ_DEV=/scratch/dingpf/docker-scratch/cvmfs_dunedaq-dev
export LOCAL_CVMFS_DUNEDAQ=/scratch/dingpf/docker-scratch/cvmfs_dunedaq
mkdir -p $LOCAL_CVMFS_DUNEDAQ/spack-releases

docker run --rm -it --net=host \
-v $LOCAL_CVMFS_DUNEDAQ/spack-releases:/cvmfs/dunedaq.opensciencegrid.org/spack-releases \
-v /cvmfs/dunedaq.opensciencegrid.org/spack-externals:/cvmfs/dunedaq.opensciencegrid.org/spack-externals \
-v $LOCAL_CVMFS_DUNEDAQ_DEV:/cvmfs/dunedaq-development.opensciencegrid.org \
dunedaq/sl7-minimal:spack

Setting up the directories

For sandbox:

export SPACK_RELEASE=/cvmfs/dunedaq-development.opensciencegrid.org/sandbox/spack-nightly
export SPACK_EXTERNALS=/cvmfs/dunedaq-development.opensciencegrid.org/sandbox/spack-externals
# the next line after this comment can be replaced with:
# export DAQ_RELEASE=N$(date +%y-%m-%d)
export DAQ_RELEASE=N22-04-02
mkdir -p ${SPACK_RELEASE}/${DAQ_RELEASE}

For production:

export SPACK_RELEASE=/cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly
export SPACK_EXTERNALS=/cvmfs/dunedaq.opensciencegrid.org/spack-externals
# the next line after this comment can be replaced with:
# export DAQ_RELEASE=N$(date +%y-%m-%d)
export DAQ_RELEASE=N22-04-02
mkdir -p ${SPACK_RELEASE}/${DAQ_RELEASE}

Install a spack instance

cd ${SPACK_RELEASE}/${DAQ_RELEASE}
export http_proxy=http://np04-web-proxy.cern.ch:3128
export https_proxy=http://np04-web-proxy.cern.ch:3128
wget https://github.com/spack/spack/archive/refs/tags/v0.17.1.tar.gz
tar xf v0.17.1.tar.gz
ln -s spack-0.17.1 spack-installation
rm -f v0.17.1.tar.gz

Generating spack repo for a nightly release

## load python3.8.3 and py-pyyaml from externals instance
source $SPACK_EXTERNALS/spack-0.17.1/share/spack/setup-env.sh
spack load py-pyyaml@5.3.1

export DAQ_RELEASE_DIR=$HOME/daq-release

git clone https://github.com/DUNE-DAQ/daq-release $DAQ_RELEASE_DIR -b dingpf/spack-release

cd $DAQ_RELEASE_DIR
python3 scripts/spack/make-release-repo.py -u \
-i configs/dunedaq-develop/dunedaq-develop.yaml \
-t spack-repos/release-repo-template \
-r ${DAQ_RELEASE} \
-o ${SPACK_RELEASE}/${DAQ_RELEASE}

Modify spack default settings to use the `externals` upstream spack instance

🔴 Note: code blocks below does not run in batch mode, it requires modifying a file/files interactively. 🔴

cd $SPACK_RELEASE/$DAQ_RELEASE
source spack-0.17.1/share/spack/setup-env.sh
spack repo add $SPACK_EXTERNALS/spack-repo-externals
cat ~/.spack/repos.yaml >>$SPACK_ROOT/etc/spack/defaults/repos.yaml

# modify repos.yaml, make sure built-in repo is at the end of the list.
vi $SPACK_ROOT/etc/spack/defaults/repos.yaml
rm -f ~/.spack/repos.yaml

# verify the spack repo has been added
spack repo list

# adding externals spack instance as an upstream

cat <<EOT >> $SPACK_ROOT/etc/spack/defaults/upstreams.yaml
upstreams:
  spack-externals:
    install_tree: ${SPACK_EXTERNALS}/spack-0.17.1/opt/spack
EOT

# run "spack find" to make sure the externals insance is successfully added
spack find

# Adding compilers installed in the externals instance to "compilers.yaml"
# We can do this by simply copying over the "compilers.yaml" in the externlas instance
cp $SPACK_EXTERNALS/spack-0.17.1/etc/spack/defaults/linux/compilers.yaml \
$SPACK_RELEASE/$DAQ_RELEASE/spack-0.17.1/etc/spack/defaults/linux/
# verify the compilers have been added
spack compiler list

# modify the installation prefix for spack packages
# here we simply copy over the same settings as the externals instance
# use "\cp" as in RHEL, "cp" is an alias of "cp -i"; "\cp" will do "unalias".
\cp $SPACK_EXTERNALS/spack-0.17.1/etc/spack/defaults/config.yaml  $SPACK_RELEASE/$DAQ_RELEASE/spack-0.17.1/etc/spack/defaults/config.yaml

Install the release in a spack environment

# spack does not allow env names contain any dots 
export DUNEDAQ_SPACK_ENV=${DAQ_RELEASE//./-}
spack env create $DUNEDAQ_SPACK_ENV
spack env activate $DUNEDAQ_SPACK_ENV -p

# adding the release repo
spack repo add $SPACK_RELEASE/$DAQ_RELEASE/spack-repo

# add the release umbrella package to the spack environment
spack add dunedaq@${DAQ_RELEASE}%gcc@8.2.0

# run spack concretize to ensure the concretizer gets a successfully result before installation
spack concretize

# install the release
spack install

If the steps above were executed successfully, assuming there is no changes to the set of external packages, you will see spack starts to install DAQ packages ONLY without any external dependencies at this moment. All external dependencies should have been found in the externals instance, and spack is smart enough to reuse them.

Prepare the prebuilt python virtual environment `dbt-pyvenv` for a release

In summary, you can follow the instructions to use the current cvmfs production deployment in this wiki page](https://github.com/DUNE-DAQ/daq-release/wiki/Instructions-to-use-the-spack-releases-deployed-in-cvmfs) and then create the dbt-pyvenv in the release directory.

Once the umbrella package dunedaq for the release is loaded, you can proceed with creating a python virtual environment with the pyvenv_requirements.txt found in the release configuration directory in daq-release repo:

daq-release/configs/dunedaq-vX.X.X/pyvenv_requirements.txt for frozen releases;
daq-release/configs/dunedaq-develop/pyvenv_requirements.txt for nightly releases.

❤️ You can also refer to the detailed steps in the sections below. ❤️

Firstly, start up the docker container with the locally prepared cvmfs deployment.

# Change the following two envs to different path 
# if you are not the owner of `/scratch/dingpf` directory. :)
export LOCAL_CVMFS_DUNEDAQ_DEV=/scratch/dingpf/docker-scratch/cvmfs_dunedaq-dev
export LOCAL_CVMFS_DUNEDAQ=/scratch/dingpf/docker-scratch/cvmfs_dunedaq

# sync pypi-repo to local cvmfs area
rsync -rlpt /cvmfs/dunedaq.opensciencegrid.org/pypi-repo $LOCAL_CVMFS_DUNEDAQ

docker run --rm -it --net=host \
-v $LOCAL_CVMFS_DUNEDAQ:/cvmfs/dunedaq.opensciencegrid.org \
-v $LOCAL_CVMFS_DUNEDAQ_DEV:/cvmfs/dunedaq-development.opensciencegrid.org \
dunedaq/sl7-minimal:spack

Secondly, setting up some ENVs according to the cvmfs deployment and targeting release:

🔴 Don't copy&run the whole code block, choose one of the four cases below. 🔴

# For nightly release in sandbox
export SPACK_RELEASE=/cvmfs/dunedaq-development.opensciencegrid.org/sandbox/spack-nightly
export DAQ_RELEASE=N22-04-02

# For forzen release in sandbox
export SPACK_RELEASE=/cvmfs/dunedaq-development.opensciencegrid.org/sandbox/spack-releases
export DAQ_RELEASE=dunedaq-v2.10.1

# For nightly release in production
export SPACK_RELEASE=/cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly
export DAQ_RELEASE=N22-04-02

# For forzen release in production
export SPACK_RELEASE=/cvmfs/dunedaq.opensciencegrid.org/spack-releases
export DAQ_RELEASE=dunedaq-v2.10.1

Thirdly, run the following to create `dbt-pyvenv`

🔴 Copy&Paste for frozen release only 🔴

source $SPACK_RELEASE/$DAQ_RELEASE/spack-0.17.1/share/spack/setup-env.sh
export DUNEDAQ_SPACK_ENV=${DAQ_RELEASE//./-}
spack env activate $DUNEDAQ_SPACK_ENV -p
spack load dunedaq@${DAQ_RELEASE}

python3.8 -m venv $SPACK_RELEASE/$DAQ_RELEASE/dbt-pyvenv 
source $SPACK_RELEASE/$DAQ_RELEASE/dbt-pyvenv/bin/activate

export http_proxy=http://np04-web-proxy.cern.ch:3128
export https_proxy=http://np04-web-proxy.cern.ch:3128

export DAQ_RELEASE_DIR=$HOME/daq-release
git clone https://github.com/DUNE-DAQ/daq-release $DAQ_RELEASE_DIR -b dingpf/spack-release


# for frozen release
python3.8 -m pip install -r $DAQ_RELEASE_DIR/configs/$DAQ_RELEASE/pyvenv_requirements.txt
# copy over the other two files needed by the release from "daq-release/configs".
cp $DAQ_RELEASE_DIR/configs/$DAQ_RELEASE/pyvenv_requirements.txt $SPACK_RELEASE/$DAQ_RELEASE
cp $DAQ_RELEASE_DIR/configs/$DAQ_RELEASE/dbt-build-order.cmake $SPACK_RELEASE/$DAQ_RELEASE

🟥 For nightly releases, replace the last three lines with 🟥

# for nightly release
python3.8 -m pip install -r $DAQ_RELEASE_DIR/configs/dunedaq-develop/pyvenv_requirements.txt
cp $DAQ_RELEASE_DIR/configs/dunedaq-develop/pyvenv_requirements.txt $SPACK_RELEASE/$DAQ_RELEASE
cp $DAQ_RELEASE_DIR/configs/dunedaq-develop/dbt-build-order.cmake $SPACK_RELEASE/$DAQ_RELEASE

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preparing the deployment of DUNE DAQ spack releases to cvmfs

Making the `externals` spack instance

Setting up the directories

Copy over the `externals` repo

Generate a nightly release repo

Install a spack instance

Install `externals` repo

1. Add externals and nightly release repo to spack, modofy default `repos.yaml`

2. Install `gcc@8.2.0`, modify default `compilers.yaml`

3. Install the release umbrella package for the nightly release

4. Remove all DAQ packages by uninstall `daq-cmake` and its dependents

Making a spack instance for a single frozen release

Setting up the directories

Install a spack instance

Generating spack repo for a frozen release

Modify spack default settings to use the `externals` upstream spack instance

Install the release in a spack environment

Making a spack instance for a single nightly release

Setting up the directories

Install a spack instance

Generating spack repo for a nightly release

Modify spack default settings to use the `externals` upstream spack instance

Install the release in a spack environment

Prepare the prebuilt python virtual environment `dbt-pyvenv` for a release

Firstly, start up the docker container with the locally prepared cvmfs deployment.

Secondly, setting up some ENVs according to the cvmfs deployment and targeting release:

Thirdly, run the following to create `dbt-pyvenv`

Clone this wiki locally

Preparing the deployment of DUNE DAQ spack releases to cvmfs

Making the externals spack instance

Setting up the directories

Copy over the externals repo

Generate a nightly release repo

Install a spack instance

Install externals repo

1. Add externals and nightly release repo to spack, modofy default repos.yaml

2. Install gcc@8.2.0, modify default compilers.yaml

3. Install the release umbrella package for the nightly release

4. Remove all DAQ packages by uninstall daq-cmake and its dependents

Making a spack instance for a single frozen release

Setting up the directories

Install a spack instance

Generating spack repo for a frozen release

Modify spack default settings to use the externals upstream spack instance

Install the release in a spack environment

Making a spack instance for a single nightly release

Setting up the directories

Install a spack instance

Generating spack repo for a nightly release

Modify spack default settings to use the externals upstream spack instance

Install the release in a spack environment

Prepare the prebuilt python virtual environment dbt-pyvenv for a release

Firstly, start up the docker container with the locally prepared cvmfs deployment.

Secondly, setting up some ENVs according to the cvmfs deployment and targeting release:

Thirdly, run the following to create dbt-pyvenv

Clone this wiki locally

Making the `externals` spack instance

Copy over the `externals` repo

Install `externals` repo

1. Add externals and nightly release repo to spack, modofy default `repos.yaml`

2. Install `gcc@8.2.0`, modify default `compilers.yaml`

4. Remove all DAQ packages by uninstall `daq-cmake` and its dependents

Modify spack default settings to use the `externals` upstream spack instance

Modify spack default settings to use the `externals` upstream spack instance

Prepare the prebuilt python virtual environment `dbt-pyvenv` for a release

Thirdly, run the following to create `dbt-pyvenv`