diff --git a/content/index.md b/content/index.md new file mode 100644 index 0000000..c17774b --- /dev/null +++ b/content/index.md @@ -0,0 +1,140 @@ +# Reproducible research software development using Python + + +## Big-picture goal + +This is a hands-on course on research software engineering. Since most +participants use Python in their work, we will use Python as the example +language. We will work with an example project and go through all the steps of +a typical software project. Once we have seen the building blocks, we will try +to apply them to own projects. Workshop participants will receive and also +learn to give constructive code feedback. + + +## Prerequisites + +:::{prereq} Preparation +1. Get a **GitHub account** following [these instructions](https://coderefinery.github.io/installation/github/). +1. You will need a **text editor**. If you don't have a favorite one, we recommend + [VS Code](https://coderefinery.github.io/installation/vscode/). +1. **If you prefer to work in the terminal** and not in VS Code, set up these two (skip this if you use VS Code): + - [Git in the terminal](https://coderefinery.github.io/installation/git-in-terminal/) + - [SSH or HTTPS connection to GitHub from terminal](https://coderefinery.github.io/installation/ssh/) +1. **One of these two software environments** (if you are not sure which one to + choose or have no preference, choose Conda): + - {ref}`conda` + - {ref}`venv` (Snakemake is not available in this environment) +1. **Optional** and only on Linux: [Apptainer](https://apptainer.org/) following + [these instructions](https://apptainer.org/docs/admin/1.3/installation.html#install-from-pre-built-packages). +::: + + +## Schedule + +:::{note} +The schedule will very soon contain links to lesson material and exercises. +::: + + +### Day 1 (Sep 16) + +- 13:00-13:30 (0.5h) - **Welcome and introduction** + - Motivation (reproducibility, robustness, distribution, improvement, trust, etc.) + - Practical information (tools, communication, breaks, etc.) + - What will learn and achieve from this course? + - Example Python project we will use and develop + +- 13:30-14:45 (1.25h) - **Introduction to version control with Git and GitHub (1/2)** + - Creating a repository and porting your project to Git and GitHub + - Basic commands + +- 15:00-16:30 (1.5h) - **Introduction to version control with Git and GitHub (2/2)** + - Branching and merging + - Recovering from typical mistakes + +- 16:45-18:00 (1.25h) - **Documentation** + - In-code documentation including docstrings + - Writing good README files + - Markdown + - Sphinx + - Building documentation with GitHub Actions + - Jupyter Notebooks + + +### Day 2 (Sep 17) + +- 09:00-10:30 (1.5h) - **Collaborative version control and code review (1/2)** + - Practice code review using issues and pull requests + - Forking workflow + - Contributing changes to projects of others + +- 10:45-12:15 (1.5h) - **Collaborative version control and code review (2/2)** + - Organization strategies + - Merge vs. rebase + - Conflict resolution + +- 16:45-18:00 (1.25h) - **Debriefing and Q&A** + - Participants work on their projects + - Together we study actual codes that participants wrote or work on + - Constructively we discuss possible improvements + - Give individual feedback on code projects + + +### Day 3 (Sep 18) + +- 09:00-10:30 (1.5h) - **Automated testing** + - Unit tests + - End-to-end tests + - pytest + - GitHub Actions + +- 10:45-12:15 (1.5h) - **How to make the project more reusable?** + - Tracking dependencies with requirements.txt and environment.yml + - Recording environments in containers + +- 13:00-14:45 (1.75h) - **Code quality and good practices** + - Naming (and other) conventions, project organization, modularity + - Refactoring (explained through examples) + - Design patterns: functional design vs. object-oriented design + - How to design your code before writing it + - Structuring larger software projects in a modular way + - Command-line interfaces + - Workflows with Snakemake + +- 15:00-16:30 (1.5h) - **How to release and publish your code** + - Packaging the code + - Licenses + - Sharing the code via PyPI + - Publishing the code via Zenodo + +- 16:45-18:00 (1.25h) - **Debriefing and Q&A** + - Participants work on their projects + - Together we study actual codes that participants wrote or work on + - Constructively we discuss possible improvements + - Give individual feedback on code projects + + +### Extra material if we have time + +- Profiling memory and CPU usage +- Strategies for parallelization + + +```{toctree} +:maxdepth: 1 +:caption: Software environment +:hidden: + +installation/conda +installation/virtual-environment +``` + +```{toctree} +:maxdepth: 1 +:caption: Reference +:hidden: + +All lessons +CodeRefinery +Reusing +``` diff --git a/content/installation/conda.md b/content/installation/conda.md new file mode 100644 index 0000000..8c55ba4 --- /dev/null +++ b/content/installation/conda.md @@ -0,0 +1,87 @@ +(conda)= + +# Conda environment + +A Conda environment is an isolated software environment that is used to manage dependencies for a project +and you decide where it is located. + +You will need a `environment.yml` file that documents the dependencies: +```yaml +name: coderefinery +channels: + - conda-forge + - bioconda +dependencies: + - python >= 3.10 + - black + - click + - flit + - ipywidgets + - isort + - jupyterlab + - jupyterlab_code_formatter + - jupyterlab-git + - matplotlib + - myst-parser + - nbdime + - numpy + - pandas + - pytest + - pytest-cov + - scalene + - seaborn + - snakemake-minimal + - sphinx + - sphinx-autoapi + - sphinx-autobuild + - sphinx_rtd_theme >= 2.0 + - vulture + - scikit-image +``` + + +## Before you create a virtual environment + +1. Create a new directory for this course. +1. In this directory, create an `environment.yml` file and copy-paste the dependencies above into it. + + +## Choose the tool to manage the environment + +If you are already using one of these tools, please continue using the tool that you like and know. +If you are new to this, **we recommend using Miniconda or Miniforge**. + +- [Anaconda](https://docs.anaconda.com/anaconda/install/) + - Advantages: easy to install, easy to use, good for beginners + - Disadvantages: large download, installs more than we will need, license restrictions +- [Miniconda](https://docs.anaconda.com/miniconda/) + - Advantages: small size, installs only what you need + - Disadvantages: no graphical interface, license restrictions +- [Miniforge](https://github.com/conda-forge/miniforge) + - Advantages: small size, no license restrictions + - Disadvantages: no graphical interface +- [Micromamba](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html) + - Advantages: fast, small size + - Disadvantages: no graphical interface +- [Pixi](https://pixi.sh/latest/) + - Advantages: fast and new + - Disadvantages: new and less tested and not documented here + + +## Creating the virtual environment + +1. Open your terminal shell (e.g. Bash or Zsh). +2. Activate `conda` using `conda activate` or `source ~/miniconda3/bin/activate`. +3. Run the following command: + ```console + $ conda env create --file environment.yml + ``` +4. Make sure that you see "coderefinery" in the output when you ask for a list of all available environments: + ```console + $ conda env list + ``` + + +## How to verify that this worked + +(this will be added) diff --git a/content/installation/virtual-environment.md b/content/installation/virtual-environment.md new file mode 100644 index 0000000..67e4961 --- /dev/null +++ b/content/installation/virtual-environment.md @@ -0,0 +1,82 @@ +(venv)= + +# Virtual environment + +A virtual environment is an isolated software environment that is used to manage dependencies for a project +and you decide where it is located. + +You will need a `requirements.txt` file that documents the dependencies: +``` +black +click +flit +ipywidgets +isort +jupyterlab +jupyterlab-code-formatter +jupyterlab-git +matplotlib +myst-parser +nbdime +numpy +pandas +pytest +pytest-cov +scalene +seaborn +sphinx +sphinx-autoapi +sphinx-autobuild +sphinx_rtd_theme >= 2.0 +vulture +scikit-image +``` + + +## Before you create a virtual environment + +1. Create a new directory for this course. +1. In this directory, create a `requirements.txt` file and copy-paste the dependencies above into it. + + +## Creating the virtual environment + +Now create a virtual environment in this directory either using [pip and +venv](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/) +(more traditional and safer) or using [uv](https://docs.astral.sh/uv/) (more modern but also less tested): + +:::::{tabs} + +::::{group-tab} pip and venv +Create a new virtual environment and activate it: +```console +$ python3 -m venv coderefinery-environment +$ source coderefinery-environment/bin/activate +``` + +Install the dependencies into the environment: +```console +$ python3 -m pip install -r requirements.txt +``` +:::: + + +::::{group-tab} uv +Create a new virtual environment and activate it: +```console +$ uv venv coderefinery-environment +$ source coderefinery-environment/bin/activate +``` + +Install the dependencies into the environment: +```console +$ uv pip install -r requirements.txt +``` +:::: + +::::: + + +## How to verify that this worked + +(this will be added)