emacsPackages: clean the bulk-updating scripts

doc/languages-frameworks/elisp.section.md

@@ -0,0 +1,75 @@
+# Emacs Lisp {#sec-emacs-lisp}
+
+Since at least version 24, Emacs has a native package manager, `package.el`. Since its inception, many Elisp packages were created by Emacs community at large, as well as package repositories.
+
+<!--
+TODO: add a chapter about melpaBuild and the general Elisp Nix framework
+-->
+
+## Bulk Update {#sec-emacs-lisp-packages-bulk-update}
+
+The chief Elisp package repositories are:
+
+- [GNU ELPA](https://elpa.gnu.org/),
+- [NonGNU ELPA](https://elpa.nongnu.org/), and
+- [MELPA](https://melpa.org/).
+
+Nixpkgs provides a comprehensive infrastucture that leverages the contents of these Elisp repositories as Nix packages. Here we will the bulk-updating automation tooling.
+
+Inside the `pkgs/applications/editors/emacs/elisp-packages` subdirectory we find the following scripts:
+
+- `update-scripts-library.sh`
+
+  This script acts as a library, containing useful functions to deal with bulk updates. It can be `source`'d in both interactive and batch environments.
+
+  This library provides the following functions:
+
+  - `download_packagesets`
+
+    This function downloads from [`nix-community` Emacs Overlay](https://github.com/nix-community/emacs-overlay) the corresponding files.
+
+    It accepts any number of arguments. The only acceptable arguments are: `elpa`, `elpa-devel`, `melpa`, `melpa-stable`, `nongnu`, `nongnu-devel`.
+
+    When an argument is not recognized, the whole function bails out with an error message.
+
+  - `test_packagesets`
+
+    This function runs a simple smoke test that instantiates the corresponding package set.
+
+    It accepts multiple arguments.
+
+    The only acceptable arguments are: `elpa`, `elpa-devel`, `melpa`, `melpa-stable`, `nongnu`, `nongnu-devel`.
+
+    When an argument is not recognized, the whole function bails out with an error message.
+
+  - `commit_packagesets`
+
+    This function commits the corresponding package set, writing a short commit message.
+
+    It accepts multiple arguments.
+
+    The only acceptable arguments are: `elpa`, `elpa-devel`, `melpa`, `melpa-stable`, `nongnu`, `nongnu-devel`.
+
+    When an argument is not recognized, the whole function bails out with an error message.
+
+- `update-package-sets`
+
+  This script updates the package sets passed to it as arguments.
+
+  It accepts multiple arguments.
+
+  The only acceptable arguments are: `elpa`, `elpa-devel`, `nongnu`, `nongnu-devel`.
+
+  When an argument is not recognized, the whole function bails out with an error message.
+
+- `update-melpa`
+
+  This script updates `recipes-archive-melpa.json`, a JSON file that describes the MELPA package set.
+
+  It ignores the arguments.
+
+- `update-from-overlay`
+
+  This script downloads all the packagesets from overlay, then tests and commits them.
+
+  It ignores the arguments.

doc/languages-frameworks/index.md

@@ -65,6 +65,7 @@ dart.section.md
 dhall.section.md
 dlang.section.md
 dotnet.section.md
+elisp.section.md
 emscripten.section.md
 gnome.section.md
 go.section.md

doc/packages/emacs.section.md

@@ -1,8 +1,16 @@
 # Emacs {#sec-emacs}
 
+[Emacs](https://www.gnu.org/software/emacs/) is the advanced, extensible, customizable, self-documenting editor.
+
+At its core is an interpreter for Emacs Lisp (shortly, Elisp), a dialect of the Lisp programming language with extensions to support text editing.
+
 ## Configuring Emacs {#sec-emacs-config}
 
-The Emacs package comes with some extra helpers to make it easier to configure. `emacs.pkgs.withPackages` allows you to manage packages from ELPA. This means that you will not have to install that packages from within Emacs. For instance, if you wanted to use `company` `counsel`, `flycheck`, `ivy`, `magit`, `projectile`, and `use-package` you could use this as a `~/.config/nixpkgs/config.nix` override:
+Nixpkgs provides a framework that leverages Emacs via Nix.
+
+`emacs.pkgs.withPackages` allows managing packages from various Elisp package repositories (ELPA, MELPA etc.) and even third-party stand-alone packages from Nix, without relying on Emacs to download and install them.
+
+For instance, by writing the code below at `~/.config/nixpkgs/config.nix`, the Elisp packages `company`, `counsel`, `flycheck`, `ivy`,`magit`, `projectile`, and `use-package` are installed:
 
 ```nix
 {
@@ -20,7 +28,15 @@ The Emacs package comes with some extra helpers to make it easier to configure.
 }
 ```
 
-You can install it like any other packages via `nix-env -iA myEmacs`. However, this will only install those packages. It will not `configure` them for us. To do this, we need to provide a configuration file. Luckily, it is possible to do this from within Nix! By modifying the above example, we can make Emacs load a custom config file. The key is to create a package that provides a `default.el` file in `/share/emacs/site-start/`. Emacs knows to load this file automatically when it starts.
+This custom package can be installed like any other one, via `nix-env -iA myEmacs`.
+
+However, this expression merely installs the package; no configuration happens here. Such a configuration requires an Elisp file crafted for this purpose.
+
+Luckily, it is possible to do this from within Nix!
+
+By modifying the above example, we can make Emacs load a custom config file. The key is to create a package that provides a `default.el` file in `/share/emacs/site-start/` -- since Emacs loads this file automatically when it starts.
+
+Let's modify the example above:
 
 ```nix
 {
@@ -97,9 +113,13 @@ You can install it like any other packages via `nix-env -iA myEmacs`. However, t
 }
 ```
 
-This provides a fairly full Emacs start file. It will load in addition to the user's personal config. You can always disable it by passing `-q` to the Emacs command.
+This provides a fairly full Emacs start file. It will be loaded in addition to the user's personal config. Further, it can be disabled at will: Emacs accepts the command-line argument `--no-init-file` (short form `-q`) to not load the init files.
+
+Sometimes `emacs.pkgs.withPackages` is not enough, as this package set imposes some priorities over their packages (with the lowest priority assigned to GNU-devel ELPA, and the highest for packages manually defined in `pkgs/applications/editors/emacs/elisp-packages/manual-packages`).
+
+However it is not possible to control these priorities when some package is installed as a dependency. In principle, the overrides can be done on a per-package-basis, providing all the required dependencies manually. However this procedure is tedious and error-prone, since there is the possibility that an unwanted dependency sneaks in through some other package.
 
-Sometimes `emacs.pkgs.withPackages` is not enough, as this package set has some priorities imposed on packages (with the lowest priority assigned to GNU-devel ELPA, and the highest for packages manually defined in `pkgs/applications/editors/emacs/elisp-packages/manual-packages`). But you can't control these priorities when some package is installed as a dependency. You can override it on a per-package-basis, providing all the required dependencies manually, but it's tedious and there is always a possibility that an unwanted dependency will sneak in through some other package. To completely override such a package, you can use `overrideScope`.
+A complete and pervasive override for such a package can be done via `overrideScope`.
 
 ```nix
 let

doc/redirects.json

@@ -3899,6 +3899,12 @@
   "sec-emacs-config": [
     "index.html#sec-emacs-config"
   ],
+  "sec-emacs-lisp": [
+    "index.html#sec-emacs-lisp"
+  ],
+  "sec-emacs-lisp-packages-bulk-update": [
+    "index.html#sec-emacs-lisp-packages-bulk-update"
+  ],
   "sec-firefox": [
     "index.html#sec-firefox"
   ],

pkgs/applications/editors/emacs/elisp-packages/elisp-archives-table.json

@@ -0,0 +1,32 @@
+{
+    "elpa": {
+        "overlay_url": "https://raw.githubusercontent.com/nix-community/emacs-overlay/master/repos/elpa/elpa-generated.nix",
+        "generated_file": "elpa-generated.nix",
+        "nix_attribute": "emacsPackages.elpaPackages"
+    },
+    "elpa-devel": {
+        "overlay_url": "https://raw.githubusercontent.com/nix-community/emacs-overlay/master/repos/elpa/elpa-devel-generated.nix",
+        "generated_file": "elpa-devel-generated.nix",
+        "nix_attribute": "emacsPackages.elpaDevelPackages"
+    },
+    "melpa": {
+        "overlay_url": "https://raw.githubusercontent.com/nix-community/emacs-overlay/master/repos/melpa/recipes-archive-melpa.json",
+        "generated_file": "recipes-archive-melpa.json",
+        "nix_attribute": "emacsPackages.melpaPackages"
+    },
+    "melpa-stable": {
+        "overlay_url": "https://raw.githubusercontent.com/nix-community/emacs-overlay/master/repos/melpa/recipes-archive-melpa.json",
+        "generated_file": "recipes-archive-melpa.json",
+        "nix_attribute": "emacsPackages.melpaStablePackages"
+    },
+    "nongnu": {
+        "overlay_url": "https://raw.githubusercontent.com/nix-community/emacs-overlay/master/repos/nongnu/nongnu-generated.nix",
+        "generated_file": "nongnu-generated.nix",
+        "nix_attribute": "emacsPackages.nongnuPackages"
+    },
+    "nongnu-devel": {
+        "overlay_url": "https://raw.githubusercontent.com/nix-community/emacs-overlay/master/repos/nongnu/nongnu-devel-generated.nix",
+        "generated_file": "nongnu-devel-generated.nix",
+        "nix_attribute": "emacsPackages.nongnuDevelPackages"
+    }
+}

pkgs/applications/editors/emacs/elisp-packages/elpa-devel-packages.nix

@@ -1,27 +1,3 @@
-/*
-
-# Updating
-
-To update the list of packages from ELPA,
-
-1. Run `./update-elpa-devel`.
-2. Check for evaluation errors:
-     # "../../../../../" points to the default.nix from root of Nixpkgs tree
-     env NIXPKGS_ALLOW_BROKEN=1 nix-instantiate ../../../../../ -A emacs.pkgs.elpaDevelPackages
-3. Run `git commit -m "elpa-devel-packages $(date -Idate)" -- elpa-devel-generated.nix`
-
-## Update from overlay
-
-Alternatively, run the following command:
-
-./update-from-overlay
-
-It will update both melpa and elpa packages using
-https://github.com/nix-community/emacs-overlay. It's almost instantaneous and
-formats commits for you.
-
-*/
-
 { lib, pkgs, buildPackages }:
 
 self: let

pkgs/applications/editors/emacs/elisp-packages/elpa-packages.nix

@@ -1,27 +1,3 @@
-/*
-
-# Updating
-
-To update the list of packages from ELPA,
-
-1. Run `./update-elpa`.
-2. Check for evaluation errors:
-     # "../../../../../" points to the default.nix from root of Nixpkgs tree
-     env NIXPKGS_ALLOW_BROKEN=1 nix-instantiate ../../../../../ -A emacs.pkgs.elpaPackages
-3. Run `git commit -m "elpa-packages $(date -Idate)" -- elpa-generated.nix`
-
-## Update from overlay
-
-Alternatively, run the following command:
-
-./update-from-overlay
-
-It will update both melpa and elpa packages using
-https://github.com/nix-community/emacs-overlay. It's almost instantaneous and
-formats commits for you.
-
-*/
-
 { lib, pkgs, buildPackages }:
 
 self: let

pkgs/applications/editors/emacs/elisp-packages/melpa-packages.nix

@@ -1,28 +1,3 @@
-/*
-
-# Updating
-
-To update the list of packages from MELPA,
-
-1. Run `./update-melpa`
-2. Check for evaluation errors:
-     # "../../../../../" points to the default.nix from root of Nixpkgs tree
-     env NIXPKGS_ALLOW_BROKEN=1 nix-instantiate --show-trace ../../../../../ -A emacs.pkgs.melpaStablePackages
-     env NIXPKGS_ALLOW_BROKEN=1 nix-instantiate --show-trace ../../../../../ -A emacs.pkgs.melpaPackages
-3. Run `git commit -m "melpa-packages $(date -Idate)" recipes-archive-melpa.json`
-
-## Update from overlay
-
-Alternatively, run the following command:
-
-./update-from-overlay
-
-It will update both melpa and elpa packages using
-https://github.com/nix-community/emacs-overlay. It's almost instantenous and
-formats commits for you.
-
-*/
-
 let
   # Read ./recipes-archive-melpa.json in an outer let to make sure we only do this once.
   defaultArchive = builtins.fromJSON (builtins.readFile ./recipes-archive-melpa.json);

pkgs/applications/editors/emacs/elisp-packages/nongnu-devel-packages.nix

@@ -1,15 +1,3 @@
-/*
-  # Updating
-
-  To update the list of packages from nongnu devel (ELPA),
-
-  1. Run `./update-nongnu-devel`.
-  2. Check for evaluation errors:
-       # "../../../../../" points to the default.nix from root of Nixpkgs tree
-       env NIXPKGS_ALLOW_BROKEN=1 nix-instantiate ../../../../../ -A emacs.pkgs.nongnuDevelPackages
-  3. Run `git commit -m "nongnu-devel-packages $(date -Idate)" -- nongnu-devel-generated.nix`
-*/
-
 {
   lib,
   pkgs,

pkgs/applications/editors/emacs/elisp-packages/nongnu-packages.nix

@@ -1,17 +1,3 @@
-/*
-
-# Updating
-
-To update the list of packages from nongnu (ELPA),
-
-1. Run `./update-nongnu`.
-2. Check for evaluation errors:
-     # "../../../../../" points to the default.nix from root of Nixpkgs tree
-     env NIXPKGS_ALLOW_BROKEN=1 nix-instantiate ../../../../../ -A emacs.pkgs.nongnuPackages
-3. Run `git commit -m "nongnu-packages $(date -Idate)" -- nongnu-generated.nix`
-
-*/
-
 { lib, pkgs, buildPackages }:
 
 self: let

pkgs/applications/editors/emacs/elisp-packages/update

@@ -1,10 +1,12 @@
 #!/usr/bin/env bash
 set -euo pipefail
 
-SCRIPT_DIR="$( cd "$(dirname "$0")" ; pwd -P )"
-cd "$SCRIPT_DIR"
+# Classic "where I am" block
+# https://www.binaryphile.com/bash/2020/01/12/determining-the-location-of-your-script-in-bash.html
+SCRIPT_DIR=$(cd $(dirname ${BASH_SOURCE[0]}); cd -P $(dirname $(readlink ${BASH_SOURCE[0]} || echo .)); pwd)
+cd "${SCRIPT_DIR}"
 
 ./update-from-overlay
 
 ./update-manual
-git commit -m "emacs.pkgs.manualPackages: $(date --iso)" -- .
+git commit -m "emacsPackages.manualPackages: updated at $(date --iso) (non-interactively)" -- .

pkgs/applications/editors/emacs/elisp-packages/update-from-overlay

@@ -1,49 +1,19 @@
 #!/usr/bin/env nix-shell
 #! nix-shell -i bash -p curl nix coreutils
-set -euxo pipefail
+set -euo pipefail
 
-# This script piggybacks on the automatic code generation done by the nix-community emacs overlay
-# You can use this to avoid running lengthy code generation jobs locally
+# This script relies on the automatic code generation done by the emacs overlay
+# kept by nix-community
+# You can use this to avoid running time-consuming code generation jobs locally
 
-export NIXPKGS_ALLOW_BROKEN=1
+source ./update-scripts-library.sh
 
-download_change() {
-    local FILE_LOCATION="$1"
+# The loops are unrolled as follows because either all packagesets should be
+# updated, or none of them should. Therefore, if any of them fails, all the
+# process should be aborted as soon as possible.
 
-    local BASEURL="https://raw.githubusercontent.com/nix-community/emacs-overlay/master/repos"
+download_packagesets "elpa" "elpa-devel" "melpa" "nongnu" "nongnu-devel"
 
-    curl -s -O "${BASEURL}/${FILE_LOCATION}"
-}
+test_packagesets "elpa" "elpa-devel" "melpa" "melpa-stable" "nongnu" "nongnu-devel"
 
-commit_change() {
-    local MESSAGE="$1"
-    local FILENAME="$2"
-
-    git diff --exit-code "${FILENAME}" > /dev/null || \
-        git commit -m "${MESSAGE}: updated $(date --iso) (from overlay)" -- "${FILENAME}"
-}
-
-test_packageset(){
-    local PKGSET="$1"
-
-    nix-instantiate --show-trace ../../../../../ -A "emacs.pkgs.$PKGSET"
-}
-
-download_change "elpa/elpa-generated.nix"
-download_change "elpa/elpa-devel-generated.nix"
-download_change "melpa/recipes-archive-melpa.json"
-download_change "nongnu/nongnu-generated.nix"
-download_change "nongnu/nongnu-devel-generated.nix"
-
-test_packageset "nongnuPackages"
-test_packageset "nongnuDevelPackages"
-test_packageset "elpaPackages"
-test_packageset "elpaDevelPackages"
-test_packageset "melpaStablePackages"
-test_packageset "melpaPackages"
-
-commit_change "elpa-packages" "elpa-generated.nix"
-commit_change "elpa-devel-packages" "elpa-devel-generated.nix"
-commit_change "melpa-packages" "recipes-archive-melpa.json"
-commit_change "nongnu-packages" "nongnu-generated.nix"
-commit_change "nongnu-devel-packages" "nongnu-devel-generated.nix"
+commit_packagesets "elpa" "elpa-devel" "melpa" "nongnu" "nongnu-devel"