Explanations
+Explanation of how the library works and why it works that way.
+ + +diff --git a/2024.8.1gui/.buildinfo b/2024.8.1gui/.buildinfo new file mode 100644 index 0000000..dbd0560 --- /dev/null +++ b/2024.8.1gui/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: 9b5ca99a40dface56730f10febf94955 +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/2024.8.1gui/.doctrees/environment.pickle b/2024.8.1gui/.doctrees/environment.pickle new file mode 100644 index 0000000..0a7ea38 Binary files /dev/null and b/2024.8.1gui/.doctrees/environment.pickle differ diff --git a/2024.8.1gui/.doctrees/explanations.doctree b/2024.8.1gui/.doctrees/explanations.doctree new file mode 100644 index 0000000..0acdd93 Binary files /dev/null and b/2024.8.1gui/.doctrees/explanations.doctree differ diff --git a/2024.8.1gui/.doctrees/explanations/differences.doctree b/2024.8.1gui/.doctrees/explanations/differences.doctree new file mode 100644 index 0000000..f93a94a Binary files /dev/null and b/2024.8.1gui/.doctrees/explanations/differences.doctree differ diff --git a/2024.8.1gui/.doctrees/explanations/how_it_works.doctree b/2024.8.1gui/.doctrees/explanations/how_it_works.doctree new file mode 100644 index 0000000..28e7d7d Binary files /dev/null and b/2024.8.1gui/.doctrees/explanations/how_it_works.doctree differ diff --git a/2024.8.1gui/.doctrees/explanations/why_dev_c7.doctree b/2024.8.1gui/.doctrees/explanations/why_dev_c7.doctree new file mode 100644 index 0000000..69ed13f Binary files /dev/null and b/2024.8.1gui/.doctrees/explanations/why_dev_c7.doctree differ diff --git a/2024.8.1gui/.doctrees/how-to.doctree b/2024.8.1gui/.doctrees/how-to.doctree new file mode 100644 index 0000000..83bc767 Binary files /dev/null and b/2024.8.1gui/.doctrees/how-to.doctree differ diff --git a/2024.8.1gui/.doctrees/how-to/containers.doctree b/2024.8.1gui/.doctrees/how-to/containers.doctree new file mode 100644 index 0000000..209b9cf Binary files /dev/null and b/2024.8.1gui/.doctrees/how-to/containers.doctree differ diff --git a/2024.8.1gui/.doctrees/how-to/deriving.doctree b/2024.8.1gui/.doctrees/how-to/deriving.doctree new file mode 100644 index 0000000..bc7ed5b Binary files /dev/null and b/2024.8.1gui/.doctrees/how-to/deriving.doctree differ diff --git a/2024.8.1gui/.doctrees/how-to/podman.doctree b/2024.8.1gui/.doctrees/how-to/podman.doctree new file mode 100644 index 0000000..2d37ef0 Binary files /dev/null and b/2024.8.1gui/.doctrees/how-to/podman.doctree differ diff --git a/2024.8.1gui/.doctrees/how-to/troubleshooting.doctree b/2024.8.1gui/.doctrees/how-to/troubleshooting.doctree new file mode 100644 index 0000000..37caf14 Binary files /dev/null and b/2024.8.1gui/.doctrees/how-to/troubleshooting.doctree differ diff --git a/2024.8.1gui/.doctrees/index.doctree b/2024.8.1gui/.doctrees/index.doctree new file mode 100644 index 0000000..c964d65 Binary files /dev/null and b/2024.8.1gui/.doctrees/index.doctree differ diff --git a/2024.8.1gui/.doctrees/reference.doctree b/2024.8.1gui/.doctrees/reference.doctree new file mode 100644 index 0000000..8b1ce98 Binary files /dev/null and b/2024.8.1gui/.doctrees/reference.doctree differ diff --git a/2024.8.1gui/.doctrees/reference/contributing.doctree b/2024.8.1gui/.doctrees/reference/contributing.doctree new file mode 100644 index 0000000..848505e Binary files /dev/null and b/2024.8.1gui/.doctrees/reference/contributing.doctree differ diff --git a/2024.8.1gui/.doctrees/tutorials.doctree b/2024.8.1gui/.doctrees/tutorials.doctree new file mode 100644 index 0000000..627d1fc Binary files /dev/null and b/2024.8.1gui/.doctrees/tutorials.doctree differ diff --git a/2024.8.1gui/.doctrees/tutorials/continue.doctree b/2024.8.1gui/.doctrees/tutorials/continue.doctree new file mode 100644 index 0000000..ce4a7bb Binary files /dev/null and b/2024.8.1gui/.doctrees/tutorials/continue.doctree differ diff --git a/2024.8.1gui/.doctrees/tutorials/prompt.doctree b/2024.8.1gui/.doctrees/tutorials/prompt.doctree new file mode 100644 index 0000000..2ca06a6 Binary files /dev/null and b/2024.8.1gui/.doctrees/tutorials/prompt.doctree differ diff --git a/2024.8.1gui/.doctrees/tutorials/start.doctree b/2024.8.1gui/.doctrees/tutorials/start.doctree new file mode 100644 index 0000000..682ebed Binary files /dev/null and b/2024.8.1gui/.doctrees/tutorials/start.doctree differ diff --git a/2024.8.1gui/.doctrees/tutorials/vscode.doctree b/2024.8.1gui/.doctrees/tutorials/vscode.doctree new file mode 100644 index 0000000..6ce8a22 Binary files /dev/null and b/2024.8.1gui/.doctrees/tutorials/vscode.doctree differ diff --git a/2024.8.1gui/_images/button.png b/2024.8.1gui/_images/button.png new file mode 100644 index 0000000..dc17cd8 Binary files /dev/null and b/2024.8.1gui/_images/button.png differ diff --git a/2024.8.1gui/_sources/explanations.rst.txt b/2024.8.1gui/_sources/explanations.rst.txt new file mode 100644 index 0000000..888750e --- /dev/null +++ b/2024.8.1gui/_sources/explanations.rst.txt @@ -0,0 +1,13 @@ +:orphan: + +Explanations +============ + +Explanation of how the library works and why it works that way. + +.. toctree:: + :caption: Explanations + + explanations/how_it_works + explanations/differences + explanations/why_dev_c7 diff --git a/2024.8.1gui/_sources/explanations/differences.rst.txt b/2024.8.1gui/_sources/explanations/differences.rst.txt new file mode 100644 index 0000000..97e3aa1 --- /dev/null +++ b/2024.8.1gui/_sources/explanations/differences.rst.txt @@ -0,0 +1,76 @@ +RHEL7 native vs dev-c7 +====================== + +The ``dev-c7`` container attempts to emulate the same environment and file systems +that a DLS RHEL7 workstation sees. Inevitably there are a few differences +which are documented on this page. + +Centos vs RHEL7 +--------------- + +This container is based upon a Centos 7 image rather than a RHEL7 image +because of complicated licensing for RHEL7 containers. In almost all +respects this should make little difference because both OSes have the +same packages available to them. + +Any code that queries the OS name will get a different answer. This affects +dls-release.py because it uses the OS name to choose the default RHEL version +to release to. However this has been patched to accept Centos 7 to mean +a release for RHEL7. + +We are not aware of any other DLS tools affected by this at present. + +User IDs +-------- + +Inside of the container there are only two interactive users. Your own user +and root. ``c7`` uses ``--userns=keep-id`` so that your same user id +is used inside and outside of the container. + +Hence the container is able to see the file ownership of mounted file systems +as long as they belong to root or to you. + +No other users are known to it and when listing the contents of +a shared directory any files that have unknown user group membership will +show up as uid or gid 65534. + +This does not affect your ability to write to files that have permission for +your secondary group. So files with group write for the dcs account will +be writeable. Unfortunately you will not be able to tell they are writeable +by using ls -l because the dcs group is unknown inside the container. + +e.g. + +.. raw:: html + +
(master) [hgv27681@dev-c7 etc]$ pwd + /dls_sw/work/R3.14.12.7/support/BL16I-BUILDER/etc + (master) [hgv27681@dev-c7 etc]$ > giles-wrote-this.txt + (master) [hgv27681@dev-c7 etc]$ ls -l + total 24 + -rw-rw-r--. 1 hgv27681 65534 0 Jun 24 10:25 giles-wrote-this.txt + -rwxrwxr-x. 1 65534 65534 1018 Feb 21 2019 home_pa_slits.py + -rw-rw-r--. 1 65534 65534 269 Feb 21 2019 Makefile + drwxrwsr-x. 5 65534 65534 16384 Jun 13 16:41 makeIocs + -rw-rw-r--. 1 65534 65534 29 Feb 21 2019 module.ini ++ + +No Services +----------- + +By default a container runs a single process (of id 1) and terminates when +that process terminates. ``c7`` launches a background process +that does nothing as process 1 and +then executes any number of interactive shells inside of it. + +Although you +have the filesystem of a Centos 7 workstation inside the +container, by design it is not an entire virtual machine. + +Therefore none of the usual services will be running inside of the container. +For example any apps that wish to communicate over DBus will not find the DBus +service. + +So far this has not affected any DLS development workflow. It may be possible +to launch services inside the container if they prove to be essential. diff --git a/2024.8.1gui/_sources/explanations/how_it_works.rst.txt b/2024.8.1gui/_sources/explanations/how_it_works.rst.txt new file mode 100644 index 0000000..dbb93ea --- /dev/null +++ b/2024.8.1gui/_sources/explanations/how_it_works.rst.txt @@ -0,0 +1,99 @@ +How the dev-c7 Container Works +============================== + + +Podman +------ + +Podman is a daemonless, rootless container engine developed by RedHat, +designed as an alternative to Docker. + +Using containers allows us to package up all the dependencies of an +application and execute it in an isolated fashion. + +In the case of ``dev-c7`` we use podman to package up everything required +by the DLS controls developer workflow. This is essentially a set +of system libraries and tools plus mounted shared file systems such +as ``/dls_sw`` and ``/home``. + +Containers +---------- + +A quote from docker.com: + +.. tip:: + + A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings. + +An image is a layered snapshot of a file system. Images adhere to the +`Open Container Initiative
' + + '' + + _("Hide Search Matches") + + "
" + ) + ); + }, + + /** + * helper function to hide the search marks again + */ + hideSearchWords: () => { + document + .querySelectorAll("#searchbox .highlight-link") + .forEach((el) => el.remove()); + document + .querySelectorAll("span.highlighted") + .forEach((el) => el.classList.remove("highlighted")); + localStorage.removeItem("sphinx_highlight_terms") + }, + + initEscapeListener: () => { + // only install a listener if it is really needed + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.shiftKey || event.altKey || event.ctrlKey || event.metaKey) return; + if (DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS && (event.key === "Escape")) { + SphinxHighlight.hideSearchWords(); + event.preventDefault(); + } + }); + }, +}; + +_ready(() => { + /* Do not call highlightSearchWords() when we are on the search page. + * It will highlight words from the *previous* search query. + */ + if (typeof Search === "undefined") SphinxHighlight.highlightSearchWords(); + SphinxHighlight.initEscapeListener(); +}); diff --git a/2024.8.1gui/_static/theme_overrides.css b/2024.8.1gui/_static/theme_overrides.css new file mode 100644 index 0000000..5fd9b72 --- /dev/null +++ b/2024.8.1gui/_static/theme_overrides.css @@ -0,0 +1,34 @@ +/* override table width restrictions */ +@media screen and (min-width: 639px) { + .wy-table-responsive table td { + /* !important prevents the common CSS stylesheets from + overriding this as on RTD they are loaded after this stylesheet */ + white-space: normal !important; + } +} + +/* override table padding */ +.rst-content table.docutils th, .rst-content table.docutils td { + padding: 4px 6px; +} + +/* Add two-column option */ +@media only screen and (min-width: 1000px) { + .columns { + padding-left: 10px; + padding-right: 10px; + float: left; + width: 50%; + min-height: 145px; + } +} + +.endcolumns { + clear: both +} + +/* Hide toctrees within columns and captions from all toctrees. + This is what makes the include trick in index.rst work */ +.columns .toctree-wrapper, .toctree-wrapper .caption-text { + display: none; +} diff --git a/2024.8.1gui/explanations.html b/2024.8.1gui/explanations.html new file mode 100644 index 0000000..f1553b5 --- /dev/null +++ b/2024.8.1gui/explanations.html @@ -0,0 +1,185 @@ + + + + + + +Explanation of how the library works and why it works that way.
+ + +The dev-c7
container attempts to emulate the same environment and file systems
+that a DLS RHEL7 workstation sees. Inevitably there are a few differences
+which are documented on this page.
This container is based upon a Centos 7 image rather than a RHEL7 image +because of complicated licensing for RHEL7 containers. In almost all +respects this should make little difference because both OSes have the +same packages available to them.
+Any code that queries the OS name will get a different answer. This affects +dls-release.py because it uses the OS name to choose the default RHEL version +to release to. However this has been patched to accept Centos 7 to mean +a release for RHEL7.
+We are not aware of any other DLS tools affected by this at present.
+Inside of the container there are only two interactive users. Your own user
+and root. c7
uses --userns=keep-id
so that your same user id
+is used inside and outside of the container.
Hence the container is able to see the file ownership of mounted file systems +as long as they belong to root or to you.
+No other users are known to it and when listing the contents of +a shared directory any files that have unknown user group membership will +show up as uid or gid 65534.
+This does not affect your ability to write to files that have permission for +your secondary group. So files with group write for the dcs account will +be writeable. Unfortunately you will not be able to tell they are writeable +by using ls -l because the dcs group is unknown inside the container.
+e.g.
+(master) [hgv27681@dev-c7 etc]$ pwd +/dls_sw/work/R3.14.12.7/support/BL16I-BUILDER/etc +(master) [hgv27681@dev-c7 etc]$ > giles-wrote-this.txt +(master) [hgv27681@dev-c7 etc]$ ls -l +total 24 +-rw-rw-r--. 1 hgv27681 65534 0 Jun 24 10:25 giles-wrote-this.txt +-rwxrwxr-x. 1 65534 65534 1018 Feb 21 2019 home_pa_slits.py +-rw-rw-r--. 1 65534 65534 269 Feb 21 2019 Makefile +drwxrwsr-x. 5 65534 65534 16384 Jun 13 16:41 makeIocs +-rw-rw-r--. 1 65534 65534 29 Feb 21 2019 module.ini +
By default a container runs a single process (of id 1) and terminates when
+that process terminates. c7
launches a background process
+that does nothing as process 1 and
+then executes any number of interactive shells inside of it.
Although you +have the filesystem of a Centos 7 workstation inside the +container, by design it is not an entire virtual machine.
+Therefore none of the usual services will be running inside of the container. +For example any apps that wish to communicate over DBus will not find the DBus +service.
+So far this has not affected any DLS development workflow. It may be possible +to launch services inside the container if they prove to be essential.
+Podman is a daemonless, rootless container engine developed by RedHat, +designed as an alternative to Docker.
+Using containers allows us to package up all the dependencies of an +application and execute it in an isolated fashion.
+In the case of dev-c7
we use podman to package up everything required
+by the DLS controls developer workflow. This is essentially a set
+of system libraries and tools plus mounted shared file systems such
+as /dls_sw
and /home
.
A quote from docker.com:
+Tip
+A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.
+An image is a layered snapshot of a file system. Images adhere to the +Open Container Initiative +standard and therefore images built by Docker and Podman are interchangeable.
+Images are stored in a container registry such as DockerHub or GitHub Container
+Registry https://ghcr.io. The image for dev-c7
is stored alongside its source
+code here:
++
The c7
script uses podman run
to create a container based on
+the above image. The container is an isolated execution environment with
+its own file system based upon the above image. Any changes to the
+file system are added in a layered fashion.
The container’s file system changes are lost when the container is deleted.
+However with c7
we arrange for that to happen only if
+the user explicitly deletes dev-c7
.
The script c7
launches a container in the background using
+podman. It then executes an interactive bash shell inside of that
+container.
This means that when you exit the bash prompt the container continues to +run in the background.
+Further invocations of c7
will execute further interactive bash
+shells in the same container.
If the container is stopped (via podman stop dev-c7
or due to a host
+reboot) then the next invocation of c7
will detect this and
+restart it.
Because of this the dev-c7
container is not ephemeral like most containers,
+it persists changes that you make in the OS until it is explicitly deleted.
The system partition in which the operating system is installed resides
+inside the container. However c7
mounts a number of host and
+shared file systems. This is how the container is made to look very
+similar to a RHEL7 workstation. The mounted file systems are as follows:
/dls_sw/prod:/dls_sw/prod
+/dls_sw/work:/dls_sw/work
+/dls_sw/epics:/dls_sw/epics
+/dls_sw/targetOS/vxWorks/Tornado-2.2:/dls_sw/targetOS/vxWorks/Tornado-2.2
+/dls_sw/apps:/dls_sw/apps
+/dls_sw/etc:/dls_sw/etc
+/scratch:/scratch
+/home:/home
+/tmp:/tmp
+/dls/science/users/:/dls/science/users/
+
c7
executes bash -l
in order to create an interactive
+shell in the container. The following features make this work:
The home directories folder /home is mounted
the HOME environment variable is passed into the container
the user namespace is mapped into the container namespace
The above points mean that bash is able to run .bash_profile
from your
+home directory under your account. Hence all the usual DLS profile
+features are loaded.
This project allows controls developers to continue to work on IOCs and tools +built for RHEL7, while at the same time allowing the DLS wide upgrade to +RHEL8 to continue.
+Some details of the plan to achieve this are here +https://confluence.diamond.ac.uk/x/NY8DCQ
+Previously the task of upgrading from RHEL6 to RHEL7 proved to be quite +complicated and took several years to finalize.
+Hence this project is a stopgap so that we can use effort to work on replacing +our +tools and deployment process with one based on containers and Kubernetes. +Some details of the epics-containers approach are documented here +https://github.com/epics-containers.
+With the new approach under development it would be a waste of effort to +port our legacy toolchain to RHEL8 only to replace it shortly afterward. +The container based approach decouples the tools and IOCs from the +host operating system so any future OS upgrades should be far +smoother.
+Practical step-by-step guides for the more experienced user.
+ + +Images are snapshots of a filesystem used to bootstrap a container.
+Containers are isolated execution environments with their file system based +upon an Image.
+You can see which Containers are running on your system with:
+podman ps -a
+
The -a
means it will also show stopped containers.
You can see what Images are cached on your system with:
+podman images
+
Images are usually cached from a container registry like ghcr.io. They can +usually be deleted without any consequence except waiting for the download +next time you launch a container that uses them.
+The container and image cache is stored in /scratch/<fed_id>/podman. If you +are short on space you can use the following.
+To remove temporary images and stopped containers:
+podman system prune
+
To remove individual images:
+podman rmi IMAGE_ID # from third column of podman images
+
+OR ...
+
+podman rmi REPOSITORY:TAG # 1st , 2nd columns of podman images
+
To remove all images with wildcard on REPOSITORY:
+podman rmi $(podman images --filter=reference='*PATTERN*' -q)
+
+# PATTERN matches the REPOSITORY column
+
The dev-c7 container image is built from the container context folder
+dev-c7/docker
.
+The context contains a Dockerfile which details the steps to build the
+container image.
To experiment with making additions to the dev-c7 project, first clone +the project and do a test build of the container:
+git clone https://dls-controls.github.io/dev-c7/main/index.html
+cd dev-c7
+./build-dev-c7
+
build-dev-c7
will create a container image with the tag
+ghcr.io/dls-controls/dev-c7
. This is the same tag that c7
+uses so it will launch a container based on your newly built image.
Note
+You will likely already have a dev-c7 container running and you must delete +that before your new one will launch. Otherwise the launch script will +just open a new bash session in the existing container. +To do this see Deleting the container.
+Now you are ready to make any changes you wish to the Dockerfile and add
+any additional files to the context (i.e. in the dev-c7/docker
folder).
Below is the current Dockerfile, read on for an explanation of the +commands used.
+# Dockerfile for getting DLS RHEL7 dev environment working in a container
+# hosted on RHEL8
+#
+# This is a stopgap so that we can use effort to promote the
+# kubernetes model https://github.com/dls-controls instead of spending
+# effort in rebuilding our toolchain for RHEL8
+
+FROM centos:7
+
+# environment
+ENV LANG=en_US.UTF-8
+ENV LC_ALL=en_US.UTF-8
+ENV LC_CTYPE=en_US.UTF-8
+ENV MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/dls_sw/apps/Modules/modulefiles:/dls_sw/etc/modulefiles:/home/hgv27681/privatemodules:/dls_sw/prod/tools/RHEL8-x86_64/defaults/modulefiles
+
+# make QT not complain
+ENV QT_X11_NO_MITSHM=1
+ENV LIBGL_ALWAYS_INDIRECT=1
+
+# switch to the vault for centos packages!
+RUN sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/*.repo && \
+ sed -i s/^#.*baseurl=http/baseurl=http/g /etc/yum.repos.d/*.repo && \
+ sed -i s/^mirrorlist=http/#mirrorlist=http/g /etc/yum.repos.d/*.repo
+
+RUN yum update -y && \
+ # dev tools and libraries
+ yum groupinstall -y "Development Tools" && \
+ yum install -y glibc.i686 redhat-lsb-core libusbx net-snmp-libs environment-modules git2u net-tools screen cmake lapack-devel readline-devel pcre-devel boost-devel libpcap-devel numactl-libs vim dbus-x11 bind-utils libssh2-devel xorg-x11-server-Xvfb \
+ # edm dependencies
+ epel-release giflib-devel libXmu-devel libpng-devel libXtst-devel zlib-devel xorg-x11-proto-devel motif-devel libX11-devel libXp-devel libXpm-devel libtirpc-devel \
+ # areaDetector dependencies
+ libxml2-devel libjpeg-turbo-devel libtiff-devel \
+ # Odin dependencies
+ hdf5-devel zeromq-devel librdkafka-devel \
+ # QT4 dependencies
+ pyqt4-devel PackageKit-gtk3-module libcanberra-gtk2 mesa-dri-drivers.x86_64 xcb-util xcb-util-devel libxcb.x86_64 libxkbcommon-x11 \
+ # dls-remote-desktop-support
+ xfreerdp zenity \
+ # useful tools
+ sudo meld tk dejavu-sans-mono-fonts gnome-terminal xterm xmessage evince eog firefox java openldap-clients && \
+ # These packages are not found in the initial install so try again
+ yum install -y zeromq-devel git2u meld && \
+ # clean up caches
+ yum clean all && \
+ # nasty hack to avoid overlay filesystem problems with --userns=keep-id
+ # hopefully can be removed when this PR is released and deployed at DLS
+ # https://github.com/containers/fuse-overlayfs/pull/381
+ rm -fr /var/lib/yum/yumdb/*
+
+# full sudo rights inside the container
+COPY /sudoers /etc/sudoers
+
+# Workaround to ensure all locales are available
+RUN sed -i "/override_install_langs=en_US/d" /etc/yum.conf
+RUN yum reinstall -y glibc-common
+
+# set container time to BST
+RUN ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
+
+# latest git from the endpoint repo https://computingforgeeks.com/install-git-2-on-centos-7/
+RUN yum remove -y git git-core && \
+ # libcurl-devel for git-remote-https
+ yum -y install libcurl-devel && \
+ yum -y install https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm && \
+ yum -y install git git-core
+
+# change the nobody account and group IDs to match RedHat
+RUN sed -i 's/99:99/65534:65534/' /etc/passwd && \
+ sed -i 's/:99:/:65534:/' /etc/group
+
The base image is selected with the FROM
command this defines what the
+container file system looks like initially. Every command in the Dockerfile
+then adds a layer of change to the container image.
The majority of the Dockerfile for dev-c7
is using yum install to get
+packages installed into the OS. Yum is executed via the RUN
command.
+Note that the RUN
command takes a single shell command line input.
Note
+It is
+common in Dockerfiles to use && \
to string multiple commands together
+in a single RUN
. The reason to do this is to minimize the number
+of layers
+in the container image. In particular, if you want to clean up temporary
+files, it must be done in the same layer they were created or it will not
+reduce the size of the resulting image (because there is a layer with the
+temp files and then another layer that removes them).
The RUN
command may execute any command that is available inside the
+container. For example the above uses a single RUN
to get the source for
+a new version of git and compile it.
Another command in this Dockerfile is ENV
which simply sets an environment
+variable for the processes the container launches.
The COPY
command allows you to supply files from the context
+(dev-c7/docker
folder in this case).
+Note that the path / is the root of the context. You can only
+copy files from inside the context, thus the context folder holds the entire
+definition of the image generated.
Each developer is free to derive their own container image if they have +specific requirements e.g.
+Additional system libraries
Additional command line or GUI tools
Any other OS configuration that they prefer
Note
+If the changes you need are likely to be useful to all developers then +you should consider contributing to the base image for dev-c7. +See Contributing
+To make your own derived container.
+Create a folder (this is your container context folder)
Make a Dockerfile in that folder
Drop in any files that you want to copy into the container
cd into the folder
podman build --tag my-dev-c7 .
c7 -i my-dev-c7
Below is an example Dockerfile derived from dev-c7 that adds a package with +yum, sets an environment variable and copies a script into /usr/bin. Note +that the build is running as root inside the container so it is allowed +to write to a system folder.
+FROM ghcr.io/dls-controls/dev-c7:latest
+
+# dev tools and libraries
+RUN yum update -y && \
+ yum install -y my-package
+
+ENV MY_VAR="useful info"
+
+COPY /my-useful-file.sh /usr/bin
+
Note
+Please ensure you are using a RHEL8 workstation before continuing.
+Previous users of podman may need to update their podman settings to +support features required by dev-c7.
+Note
+Updating your container filesystem may require deleting all images, +containers and volumes in your cache. Make sure you are able to +recreate all assets you need before continuing
+To update simply run the script /dls_sw/apps/setup-podman/setup.sh
.
+If your podman config is already up to date then this will have no effect,
+if it needs to delete your cache it will warn you before doing so.
If you do not have sudo rm permission on your workstation you will be +required to have an administrator run the script on your behalf.
+The additional features are:
+to support secondary groups in shared folders
provide a much faster container build and run
avoid loosing all scratch your files on podman system reset
If you want to use edm then you will need to have the local fonts on your +host machine. These will eventually be provided by default on the standard DLS +RHEL8 workstation. However, early adopters will need to install these +themselves.
+Use sudo yum install
on each of the rpms in this folder
+https://github.com/dls-controls/dev-c7/tree/dev/edm-fonts
When building the container for the first time, you may come across the following:
+`
+Error: writing blob: adding layer with blob
+"sha256:0b2dc63a68b9b80b6e261e0c71119894a739d353f8263d6b2f1394c66a45f5af": ApplyLayer exit status 1 stdout:
+stderr: potentially insufficient UIDs or GIDs available in user namespace (requested 0:54 for /run/lock/lockdev):
+Check /etc/subuid and /etc/subgid: lchown /run/lock/lockdev: invalid argument
+`
Rootless Podman uses a pause process to preserve the unprivileged namespaces, +which locks down the user files /etc/subuid and /etc/subgid. +The following command will stop the pause process and release the files for +editing:
+podman system migrate
+
IF you see this error:
+ERRO[0000] cannot find UID/GID for user hgv27681: No subuid ranges found for user "hgv27681" in /etc/subuid - check rootless mode in man pages.
+WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user
+
Then you probably have an empty /etc/subuid file. This is automatically updated by +cfengine at 11AM every day. If your workstation was recently built then you may +need to wait until the next 11AM !!
+Some PyQt applications may show this error:
+libGL error: unable to load driver: swrast_dri.so
+libGL error: failed to load driver: swrast
+
This is benign and can be ignored. (if the application is not launching Then +this is a different issue - don’t be distracted by this error)
+If you see this on launch of podman containers:
+/bin/sh: error while loading shared libraries: libc.so.6: cannot change memory protections
+
Then you are missing the mount_program = "/bin/fuse-overlayfs"
entry in your
+storage.conf file. See Updating Podman Settings.
A Diamond Light Source specific developer container.
+This repo defines a container and launch script for re-creating the +DLS Controls RHEL7 developer environment on a RHEL8 Workstation.
+Source code |
++ |
Documentation |
++ |
Releases |
++ |
Images |
++ |
See the tutorial at Quick Start.
+Documentation is split into four categories, also accessible from links in the +side-bar.
+Tutorials for installation, library and commandline usage. New users start here.
+Practical step-by-step guides for the more experienced user.
+ + +Explanation of how the library works and why it works that way.
+ + +Practical step-by-step guides for the more experienced user.
+ + +Practical step-by-step guides for the more experienced user.
+ + +Contributions and issues are most welcome! All issues and pull requests are +handled through GitHub. Also, please check for any existing issues before +filing a new one. If you have a great idea but it involves big changes, please +file a ticket before making a pull request! We want to make sure you don’t spend +your time coding something that might not fit the scope of the project.
+The image can be made locally with the script build-dec-c7
. The
+resulting image will be placed in the local cache with the tag
+ghcr.io/dls-controls/dev-c7:latest
.
Since this is the default tag that run-dev-c7
uses you can now use it
+launch and try out the new container image.
At present there are no tests to prove that the resulting image works. TBA.
+In order to build the documentation you will require a python virtual +environment. Set up as follows:
+$ cd dev-c7
+$ virtualenv .venv
+$ source .venv
+$ pip install -r requirements.txt
+
Documentation is contained in the docs
directory.
Docs follow the underlining convention:
+Headling 1 (page title)
+=======================
+
+Heading 2
+---------
+
+Heading 3
+~~~~~~~~~
+
You can build the docs from the project directory by running:
+$ docs/make_docs.sh
+$ firefox build/html/index.html
+
To make a new release, please follow this checklist:
+make changes on a branch and test locally with build-dev-c7
add any required updates to the docs and test with docs/make_docs.sh
push the changes to GitHub
Go to https://github.com/dls-controls/dev-c7/pulls and create a PR to main
GitHub actions will make an image to verify that it builds
Await approval for the PR
Choose a new PEP440 compliant release number
Go to release and create a new release with the new version
Check and edit for clarity the autogenerated GitHub release
Submit the release
GitHub Actions will publish the image to ghcr.io and the docs to GitHub Pages
Tutorials for installation, library and commandline usage. New users start here.
+This tutorial takes you through a few additional features provided by the +launch script.
+To see optional parameters, run the launch script with -h:
+$ c7 -h
+
+usage: c7 [options]
+
+Launches a developer container that simulates a DLS RHEL7 workstation.
+
+Options:
+
+ -h show this help
+ -l Enable logging
+ -p pull an updated version of the image first
+ -i image specify the container image (default: ghcr.io/dls-controls/dev-c7)
+ -v version specify the image version (default: latest)
+ -s host set a hostname for your container (default: pc0116.cs.diamond.ac.uk)
+ -d delete previous container and start afresh
+ -n run in podman virtual network instead of the host network
+ -c command run a command in the container (must be last option)
+ -I Install .devcontainer/devcontainer.json in the current directory for vscode
+ -g enable X11 GUI for containers launched via ssh (only required with -r)
+ -r run as root
+
The default behaviour is that c7
will use the latest container version in the
+registry on first invocation. When it launches and attaches to the
+container it will report the version you are currently using.
c7
will not pick up new releases of the
+container image unless you explicitly ask it to pull the latest again:
c7 -dp
+
If you would like to roll back to a previous version +of the container then use the -v option:
+c7 -dv 2.2.0
+
To check what versions of the image are available, take a look at the +github container registry for this project +https://ghcr.io/dls-controls/dev-c7
+Note that the container should be considered ephemeral. The system partition +can be changed and you have full sudo rights, but note:
+Changes inside the container such as yum install
will be persisted,
+but only until it is deleted.
Deletion is under your control, see below. However you will always need +to do a delete before updating to a new version. Therefore it is a bad +idea to accumulate many changes inside of the container.
If you have permanent additions to the container that you would like +to implement, see Derive your own container image
To reset the container back to its original state we ask podman +to stop it and delete it.
+Warning
+Everything running in your container will be terminated when you +do this. Make sure you have saved your work and closed all +applications.
+The following will stop the container and delete it’s state. All +dev-c7 shells and any GUI apps launched from them will be closed:
+podman rm -ft0 dev-c7
+
When you next launch the container, it will be started with its file system +initialized back to the default state specified in the image at +ghcr.io/dls-controls/dev-c7:latest.
+You can also ask c7
to perform the delete for you with -d
.
Warning
+Any changes you have made to the container itself will be lost when you
+execute the above command. This includes
+any yum install
and any changes to the operating system files.
+See How the dev-c7 Container Works for more detail. To permanently
+persist changes see Derive your own container image.
If the script has acquired new features you may want to update as follows:
+cd $HOME/bin
+rm c7
+wget -nc https://raw.githubusercontent.com/dls-controls/dev-c7/main/c7
+chmod +x c7
+
+c7 -d # -d deletes previous container to start afresh
+
Also update your devcontainer.json to match for projects you want to also +upgrade.
+It is useful to have a way to tell you are inside of the developer +container. By default your bash prompt will look exactly the +same inside and out of the container.
+You can fix this in one of two ways.
+Preferred Solution
+The file $HOME/.bashrc_local is sourced by bash for each shell that you +launch and it is a place to put your personal settings.
+Add the following to the end of the file in order to have an indicator +that you are inside the developer container.
+if [[ -f /etc/centos-release ]] ; then
+ export PS1="[C7] ${PS1}"
+fi
+
This updates the PS1 prompt that you see on the command line with a prefix
+of [C7]
.
Typically the PS1 prompt shows you which host you are on. You can change +the name of the host inside the container to give you an indicator that +you are in the developer container.
+When using the launch script for initial creation of the container you +can pass a hostname of your choice as follows.
+c7 -s my-hostname
+
Warning
+If you are working remotely and launching the container from an +ssh session changing the hostname will cause X11 apps to fail.
+First, if you have never used containers at DLS before, then you must +do an initial podman setup:
+/dls_sw/apps/setup-podman/setup.sh
+
Note
+If you have previously used podman you may need to perform a migration. +See Updating Podman Settings
+Next, copy the startup script to your local bin directory and make it +executable.
+cd $HOME/bin
+curl -O https://raw.githubusercontent.com/dls-controls/dev-c7/main/c7
+chmod +x c7
+
Finally, launch an instance of the container by typing:
+c7
+
That’s it. For further options see More Features
+Note
+if don’t have a $HOME/bin folder then you will need to do
+mkdir $HOME/bin
. It will automatically be added to your PATH in
+any new shells, but the current shell won’t see it.
Note
+For the first invocation of an updated version of the container there +will be a 30 second delay while the filesystem user id namespace mapping +is cached. This will happen after the image is pulled from the registry.
+At a dev-c7 prompt you can work normally as if you were sitting at a RHEL7 +workstation. Everything should work as before although there are a few +differences, see +RHEL7 native vs dev-c7.
+If you find anything that does not work or have suggestions for improvement, +please report it +On GitHub Issues.
+You can launch multiple instances of dev-c7 and they will share the +same container.
You can run gnome-terminal and open multiple tabs - all will be in the +container
You can use module load as usual to make further software from dls_sw +available
You can run GUI apps as normal.
You have full sudo rights and can install anything you need into the
+container with yum install
Do NOT run vscode inside the container. Instead run it normally and +feel free to launch dev-c7 inside of its integrated terminals to do +compilation, testing.
If you are using edm and the fonts look wrong or you get errors about +missing fonts, then read on. This should not be required for RHEL8 machines +at DLS which should already have the fonts installed.
+The edm display manager uses local fonts that need to be installed on your host.
+Again, these should be part of the standard RHEL8 installation but early +adopters may need to install them as follows:
+bash <(curl -s https://raw.githubusercontent.com/dls-controls/dev-c7/main/edm-fonts/install-fonts.sh)
+
VSCode has beautiful integration for developing in containers. See here +https://code.visualstudio.com/docs/remote/containers for a detailed +description of this feature.
+When running VSCode with a developer container:
+All integrated terminals run inside the container
The test explorer will run tests inside the container
File dialogs will browse the container’s file system
Python and C++ debugging are available for processes running in the +container
Run up VSCode and install the remote development plugin:
+module load vscode
+code
+
Then inside VSCode:
+Ctrl+P
+ext install ms-VSCode-remote.VSCode-remote-extensionpack
+
Add the following settings to you user configuration for VSCode in
+$HOME/.config/Code/User/settings.json
These settings ensures that each terminal loads the bash (login) profile by +default. +Bash will then run .bash_profile and this includes essential setup for +DLS controls developer environment.
+"terminal.integrated.defaultProfile.linux": "bash (login)",
+"terminal.integrated.profiles.linux": {
+ "bash (login)": {
+ "args": [
+ "-l"
+ ],
+ "path": "bash"
+ }
+}
+
This setting tells remote container development to use podman instead of +docker.
+"remote.containers.dockerPath": "podman"
+
To use VSCode with a developer container, first add the
+.devcontainer
folder with devcontainer.json
+file to the root of your project. The script has a helper to do this:
c7 -I
+
or you can create it manually:
+cd <my project folder>
+mkdir .devcontainer
+cd .devcontainer
+wget -q https://raw.githubusercontent.com/dls-controls/dev-c7/main/.devcontainer/devcontainer.json
+
Then launch VSCode:
+cd <my project folder>
+module load vscode
+code .
+
Click on Reopen in Container
when you see a dialogue with the message
+Folder contains a Dev Container configuration file.
Close the first terminal that you see and open another one (see Known Issues)
+That’s it, you are now running a developer container and your vscode session +is connected to it.
+The above gets version 2.0.0 which is current as of 01/07/2022. +See https://github.com/dls-controls/dev-c7/releases for the latest updates.
+Like the c7
script, the container will be kept alive. If you
+exit and re-enter VSCode you will be reconnected to the same container.
When you run podman ps
you will see the container is running and is named
+something like this:
localhost/vsc-dev-c7-914745539fc385a5fe9188693f0fa257-uid
+
You can recreate the container from scratch by deleting it with podman rm
+or you can tell VSCode to rebuild it using the remotes menu. This menu is
+accessed by clicking the icon in the very bottom left of the VSCode window.
The remotes menu option Rebuild Container
will close the session,
+rebuild the container
+and reopen the session in the new container.
Note that this means that the containers used by VSCode and those used
+by c7
are distinct. So if you install something into one it will
+not be seen in the other.
Note
+VSCode will use a different container for each folder. If you would +like to share a container between folders:
+open multiple folders in a single VSCode session by choosing “Add +Folder to Workspace” from the right click menu in the File Explorer
choose File->Save Workspace As...
and save the workspace to a
+file, usually in the folder that is a common root to your projects.
copy the .devcontainer
folder to the same folder as the workspace
+file.
go to the remotes menu (icon in bottom left of VSCode) and choose
+Reopen Container
If you want to remove all of the containers that VSCode has created then the +following will do the trick:
+podman rmi -f $(podman images --filter=reference='*vsc-*' -q)
+
This may be useful if you are having trouble with VSCode devcontainers or +if you are running low on disk space.
+When a devcontainer is launched, it will usually start a single terminal that is +NOT using the login profile. This means you won’t see your usual bash prompt +and none of the DLS development environment will be available.
+To work around this, close and reopen the 1st terminal +or type:
+bash -l
+