build: migrate to pyproject.toml (#1068)

Modernisation of distribution and build tooling: this PR attempts to
bring `ops` up-to-date with current packaging best practice. The
particular focus is on moving away from `setup.py` and using
`pyproject.toml` as the source of truth for configuration, with a
[PEP 517](https://peps.python.org/pep-0517/)/[PEP 518]
(https://peps.python.org/pep-0518/) compliant build system.

### Metadata

The project metadata has moved from `setup.py` to [pyproject.toml]
(pyproject.toml). This is generally a simple move, except:

* The author name and email has been updated from Charmcraft (likely a
  bad copy-and-paste) to Charm Tech.
* pyproject.toml allows for more project URLs,
  so "Homepage", "Repository", "Issues", "Documentation",
  and "Changelog" have been included.

A few `test_infra` tests were removed as a result, as they were
previously validating custom code in `setup.py` and now that
functionality is provided by the build backend (for example,
including the contents of the README).

### Source distribution changes

I've added a MANIFEST.in file to more explicitly define which files
are included. There are more files than previously, however:

* CHANGES.md
* CODE_OF_CONDUCT.md (this is more for interacting, so could be
  excluded, but it's linked from the README so seems wrong to not be
  included)
* HACKING.md
* test/bin, test/charms, test/smoke
* test/pebble_cli.py
* tox.ini

### Dependency management

The list of dependencies has moved from `setup.py` and
`requirements.txt` to `pyproject.toml`. We no longer keep two lists
of the dependencies in sync, so a `test_infra` test can be removed.
The dev requirements have been split out into groups for each tox
environment, and are located in `tox.ini`.

The documentation dependencies have moved from `docs/requirements.in`
to `pyproject.toml` in an extra-dependencies section. The
`docs/requirements.txt` file can be generated using `pip-compile`,
which removes the need for the
(undocumented) `docs/update_requirements.sh` script. `tox -e docs`
will also run the `pip-compile` step, so normally contributors should
not need to install and run `pip-compile` themselves, just do the
normal steps of running `tox -e docs` to locally inspect the docs,
and commit the updated lock file if there are changes.

If anyone is relying on `requirements.txt` or `requirements-dev.txt`
to exist (e.g. as we do with the CI that tests against key charms)
that will break, but it seems unlikely that anyone is downstream
doing that.

### CI changes

* We now verify that building and (more importantly) installing works
  on a matrix of macOS and Ubuntu in Pythons 3.8-3.12. This was
  previously only Ubuntu and only 3.11 (until recently, whatever
  Python version the GitHub Action defaulted to). It's unlikely that
  installing will break for macOS only or for specific Python
  versions, but possible.
* We now use `build` as the build frontend (`setuptools` remains the
  build backend) for building distributions to publish to PyPI
  (ideally [we have access to test.pypi.org]
  (pypi/support#3349) in order to verify
  that this works correctly before merging). Also moves back to using
  the default GitHub Action Python version, since we are no longer
  impacted by the `distutils` removal.
* Similarly, uses `build` as the build frontend for validating that
  building works correctly.

### Doc changes

* Expanded HACKING.md to include more detailed information about the
  tools that we use for development. The "Dependencies" section was
  also a sub-section of the "Documentation" section, which I think
  was an error, so promote it to top-level.

## Version

In `ops` 2.8, `ops.__version__` is:
 * `<tag>-<#commits>-g<hex>[-dirty]` (or just `<tag>` if there are no
   local commits) if there is a `.git` folder and `git
   describe --tags --dirty` runs (note that this means that importing
   `ops` from a Git clone will always spawn a `git` subprocess)
 * `1.0.dev0+unknown` if running from a non-built source (e.g. a
   GitHub tarball).
 * `<tag>` if running from a built source (e.g. from PyPI), or
   `<tag>-<#commits>-g<hex>[-dirty]` if running from a 'dirty' built
   source (e.g. a local `python setup.py sdist`) (note that this is
   from a static file generated in the build process and does not
   spawn any subprocess)

This PR replaces that with a much simpler system:

* `ops/version.py` has a static, manually managed, `version` string
  (in this module for backwards compatibility).
* Prior to publishing a release, the release manager gets a PR merged
  that sets that string to the appropriate value.
* Immediately after publishing a release, the release manager gets a
  PR merged that sets that string to the expected next release, with
  `.dev0` appended.

### Further changes

We expect to also change from using pyflakes (and extensions), isort,
and (potentially autopep8) to using ruff. Either as part of that
change, or as a further follow-up, I intend to propose an
(optional) pre-commit configuration that would optionally automatic
some of the tooling (both the formatting and linting, and also the
pip-tools management introduced in this PR).

Fixes #893, #1039

.github/workflows/framework-tests.yaml

@@ -93,20 +93,27 @@ jobs:
           PEBBLE: /tmp/pebble
 
   pip-install:
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
+
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
 
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python 3
         uses: actions/setup-python@v2
         with:
-          python-version: '3.11'
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install build dependencies
+        run: pip install wheel build
 
       - name: Build
-        run: python setup.py sdist
+        run: python -m build
 
       # Test that a pip install of the source dist .tar.gz will work
       - name: Test 'pip install'
         # Shouldn't happen, but pip install will fail if ls returns multiple lines
         run: pip install $(ls dist/ops*.gz)
-

.github/workflows/publish.yml

@@ -19,12 +19,10 @@ jobs:
       - uses: actions/checkout@v3
       - name: Setup Python
         uses: actions/setup-python@v1
-        with:
-          python-version: "3.10"
-      - name: Install wheel
-        run: pip install wheel
+      - name: Install build dependencies
+        run: pip install wheel build
       - name: Build
-        run: python setup.py sdist bdist_wheel
+        run: python -m build
       - name: Publish
         uses: pypa/gh-action-pypi-publish@release/v1
         with:

.gitignore

@@ -1,10 +1,7 @@
 __pycache__
 /sandbox
-/build
-/dist
 /ops.egg-info
 .idea
-/docs/_build
 *~
 .venv
 venv
@@ -13,6 +10,11 @@ venv
 .coverage
 /.tox
 
+# Build artifacts
+/dist
+/build
+/docs/_build
+
 # Smoke test artifacts
 *.tar.gz
 *.charm

.readthedocs.yaml

@@ -2,7 +2,10 @@ version: 2
 
 python:
   install:
-    - requirements: docs/requirements.txt
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs
 
 build:
   os: ubuntu-22.04

HACKING.md

@@ -180,26 +180,63 @@ next to the relevant content (e.g. headings, etc.).
 
 Noteworthy changes should also get a new entry in [CHANGES.md](CHANGES.md).
 
+As noted above, you can generate a local copy of the API reference docs with tox:
 
-## Dependencies
+```sh
+tox -e docs
+open docs/_build/html/index.html
+```
+
+# Dependencies
 
 The Python dependencies of `ops` are kept as minimal as possible, to avoid
 bloat and to minimise conflict with the charm's dependencies. The dependencies
-are listed in [requirements.txt](requirements.txt).
+are listed in [pyproject.toml](pyproject.toml) in the `project.dependencies` section.
+
+# Dev Tools
+
+## Formatting and Checking
 
+Test environments are managed with [tox](https://tox.wiki/) and executed with
+[pytest](https://pytest.org), with coverage measured by
+[coverage](https://coverage.readthedocs.io/).
+Static type checking is done using [pyright](https://github.com/microsoft/pyright),
+and extends the Python 3.8 type hinting support through the
+[typing_extensions](https://pypi.org/project/typing-extensions/) package.
+
+Formatting uses [isort](https://pypi.org/project/isort/) and
+[autopep8](https://pypi.org/project/autopep8/), with linting also using
+[flake8](https://github.com/PyCQA/flake8), including the
+[docstrings](https://pypi.org/project/flake8-docstrings/),
+[builtins](https://pypi.org/project/flake8-builtins/) and
+[pep8-naming](https://pypi.org/project/pep8-naming/) extensions.
+
+All tool configuration is kept in [project.toml](pyproject.toml). The list of
+dependencies can be found in the relevant `tox.ini` environment `deps` field.
+
+## Building
+
+The build backend is [setuptools](https://pypi.org/project/setuptools/), and
+the build frontend is [build](https://pypi.org/project/build/).
 
 # Publishing a Release
 
 To make a release of the ops library, do the following:
 
-1. Visit the [releases page on GitHub](https://github.com/canonical/operator/releases).
-2. Click "Draft a new release"
-3. The "Release Title" is simply the full version number, in the form <major>.<minor>.<patch>
-   E.g. 2.3.12
-4. Drop notes and a changelog in the description.
-5. When you are ready, click "Publish". (If you are not ready, click "Save as Draft".)
-
-This will trigger an automatic build for the Python package and publish it to PyPI (the API token/secret is already set up in the repository settings).
+1. Open a PR to change [version.py][ops/version.py]'s `version` to the
+   [appropriate string](https://semver.org/), and get that merged to main.
+2. Visit the [releases page on GitHub](https://github.com/canonical/operator/releases).
+3. Click "Draft a new release"
+4. The "Release Title" is simply the full version number, in the form <major>.<minor>.<patch>
+   and a brief summary of the main changes in the release
+   E.g. 2.3.12 Bug fixes for the Juju foobar feature when using Python 3.12
+5. Drop notes and a changelog in the description.
+6. When you are ready, click "Publish". (If you are not ready, click "Save as Draft".) Wait for the new version to be published successfully to [the PyPI project](https://pypi.org/project/ops/).
+7. Open a PR to change [version.py][ops/version.py]'s `version` to the expected
+   next version, with "+dev" appended (for example, if 3.14.1 is the next expected version, use
+   `'3.14.1.dev0'`).
+
+This will trigger an automatic build for the Python package and publish it to PyPI (authorization is handled via a [Trusted Publisher](https://docs.pypi.org/trusted-publishers/) relationship).
 
 See [.github/workflows/publish.yml](.github/workflows/publish.yml) for details. (Note that the versions in publish.yml refer to versions of the GitHub actions, not the versions of the ops library.)
 

MANIFEST.in

@@ -0,0 +1,10 @@
+graft ops
+graft test
+
+include *.txt
+include *.md
+include MANIFEST.in
+include pyproject.toml
+include tox.ini
+
+global-exclude *~ *.py[cod] __pycache__ *.charm

docs/requirements.txt

@@ -1,31 +1,93 @@
+#
+# This file is autogenerated by pip-compile with Python 3.8
+# by the following command:
+#
+#    pip-compile --extra=docs --output-file=docs/requirements.txt pyproject.toml
+#
 alabaster==0.7.13
-babel==2.12.1
+    # via sphinx
+babel==2.14.0
+    # via sphinx
 beautifulsoup4==4.12.2
-certifi==2023.7.22
-charset-normalizer==3.1.0
+    # via
+    #   canonical-sphinx-extensions
+    #   furo
+canonical-sphinx-extensions==0.0.18
+    # via lxd-sphinx-extensions
+certifi==2023.11.17
+    # via requests
+charset-normalizer==3.3.2
+    # via requests
 docutils==0.18.1
-furo==2023.5.20
-idna==3.4
+    # via
+    #   canonical-sphinx-extensions
+    #   sphinx
+    #   sphinx-tabs
+furo==2023.9.10
+    # via ops (pyproject.toml)
+idna==3.6
+    # via requests
 imagesize==1.4.1
-jinja2==3.1.2
-lxd-sphinx-extensions==0.0.7
-markupsafe==2.1.2
-packaging==23.1
-pygments==2.15.1
-pyyaml==6.0
+    # via sphinx
+importlib-metadata==7.0.1
+    # via sphinx
+jinja2==3.1.3
+    # via sphinx
+lxd-sphinx-extensions==0.0.16
+    # via ops (pyproject.toml)
+markupsafe==2.1.3
+    # via jinja2
+packaging==23.2
+    # via sphinx
+pygments==2.17.2
+    # via
+    #   furo
+    #   sphinx
+    #   sphinx-tabs
+pytz==2023.3.post1
+    # via babel
+pyyaml==6.0.1
+    # via ops (pyproject.toml)
 requests==2.31.0
+    # via
+    #   canonical-sphinx-extensions
+    #   sphinx
 snowballstemmer==2.2.0
-soupsieve==2.4.1
+    # via sphinx
+soupsieve==2.5
+    # via beautifulsoup4
 sphinx==6.2.1
-sphinx-basic-ng==1.0.0b1
+    # via
+    #   canonical-sphinx-extensions
+    #   furo
+    #   ops (pyproject.toml)
+    #   sphinx-basic-ng
+    #   sphinx-copybutton
+    #   sphinx-design
+    #   sphinx-tabs
+sphinx-basic-ng==1.0.0b2
+    # via furo
 sphinx-copybutton==0.5.2
-sphinx-design==0.4.1
-sphinx-tabs==3.4.1
+    # via ops (pyproject.toml)
+sphinx-design==0.5.0
+    # via ops (pyproject.toml)
+sphinx-tabs==3.4.4
+    # via ops (pyproject.toml)
 sphinxcontrib-applehelp==1.0.4
+    # via sphinx
 sphinxcontrib-devhelp==1.0.2
+    # via sphinx
 sphinxcontrib-htmlhelp==2.0.1
+    # via sphinx
 sphinxcontrib-jsmath==1.0.1
+    # via sphinx
 sphinxcontrib-qthelp==1.0.3
+    # via sphinx
 sphinxcontrib-serializinghtml==1.1.5
-urllib3==2.0.7
-websocket-client==1.5.2
+    # via sphinx
+urllib3==2.1.0
+    # via requests
+websocket-client==1.7.0
+    # via ops (pyproject.toml)
+zipp==3.17.0
+    # via importlib-metadata

ops/version.py

@@ -1,4 +1,4 @@
-# Copyright 2020 Canonical Ltd.
+# Copyright 2023 Canonical Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,45 +12,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Helper to define the version of the ops library.
+"""Package version.
 
 This module is NOT to be used when developing charms using ops.
 """
 
-import subprocess
-from pathlib import Path
-
-__all__ = ('version',)
-
-_FALLBACK = '1.0'  # this gets bumped after release
-
-
-def _get_version():
-    version = f"{_FALLBACK}.dev0+unknown"
-
-    p = Path(__file__).parent
-    if (p.parent / '.git').exists():
-        try:
-            proc = subprocess.run(
-                ['git', 'describe', '--tags', '--dirty'],
-                stdout=subprocess.PIPE,
-                stderr=subprocess.DEVNULL,
-                cwd=p,
-                check=True,
-                encoding='utf-8')
-        except Exception:
-            pass
-        else:
-            version = proc.stdout.strip()
-            if '-' in version:
-                # version will look like <tag>-<#commits>-g<hex>[-dirty]
-                # in terms of PEP 440, the tag we'll make sure is a 'public version identifier';
-                # everything after the first - needs to be a 'local version'
-                public, local = version.split('-', 1)
-                version = f"{public}+{local.replace('-', '.')}"
-                # version now <tag>+<#commits>.g<hex>[.dirty]
-                # which is PEP440-compliant (as long as <tag> is :-)
-    return version
-
-
-version = _get_version()
+version: str = '2.10.0.dev0'