From db77ad55bc6fbe2d82f1b0ec11282f041ef7a2fc Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 24 Apr 2024 23:00:36 +0530
Subject: [PATCH 01/43] Git-ignore pytest cache folders

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 6b7d99f1..c7d09ec5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -49,6 +49,7 @@ coverage.xml
 *,cover
 .hypothesis/
 cover/
+.pytest_cache/
 
 # Cython annotation files
 numcodecs/*.html

From 22cede68293f3836c94438189d13f2c791251dba Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 24 Apr 2024 23:02:50 +0530
Subject: [PATCH 02/43] Refine outdated sections for Contributing guide

This commit makes the following changes:

1. Formatting and whitespace changes
2. Removes outdated Python version references
and adds references to the current supported
version of Python for NumCodecs
3. Removes references to Travis and AppVeyor
since GitHub Actions is being used in favour of
them as a CI provider.
---
 docs/contributing.rst | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)

diff --git a/docs/contributing.rst b/docs/contributing.rst
index bb03a3ca..7987f52c 100644
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@@ -76,7 +76,7 @@ You will need your own fork to work on the code. Go to the link above and hit
 the "Fork" button. Then clone your fork to your local machine::
 
     $ git clone --recursive git@github.com:your-user-name/numcodecs.git  # with ``ssh``
-    
+
 or::
 
     $ git clone --recursive https://github.com/your-user-name/numcodecs.git  # with ``https``
@@ -97,7 +97,7 @@ you have cloned the NumCodecs source code and your current working directory is
 the repository, you can do something like the following::
 
     $ mkdir -p ~/pyenv/numcodecs-dev
-    $ virtualenv --no-site-packages --python=/usr/bin/python3.9 ~/pyenv/numcodecs-dev
+    $ virtualenv --no-site-packages --python=/usr/bin/python3.X ~/pyenv/numcodecs-dev
     $ source ~/pyenv/numcodecs-dev/bin/activate
     $ pip install -e .[docs,test,msgpack,zfpy]
 
@@ -111,7 +111,7 @@ Creating a branch
 Before you do any new work or submit a pull request, please open an issue on GitHub to
 report the bug or propose the feature you'd like to add.
 
-It's best to create a new, separate branch for each piece of work you want to do. E.g.::
+It's best to create a new, separate branch for each piece of work you want to do. E.g.:
 
     git fetch upstream
     git checkout -b shiny-new-feature upstream/main
@@ -144,12 +144,11 @@ docstrings. The simplest way to run the unit tests is to invoke::
 
     $ pytest -v
 
-NumCodecs currently supports Python 6-3.9, so the above command must
+NumCodecs currently supports Python 3.8 and later, so the above command must
 succeed before code can be accepted into the main code base.
 
-All tests are automatically run via Travis (Linux) and AppVeyor (Windows) continuous
-integration services for every pull request. Tests must pass under both services before
-code can be accepted.
+All tests are automatically run via continuous integration services for every pull request
+provided by GitHub Actions. Tests must pass under both services before code can be accepted.
 
 Code standards
 ~~~~~~~~~~~~~~
@@ -163,11 +162,11 @@ Conformance can be checked by running::
 Test coverage
 ~~~~~~~~~~~~~
 
-NumCodecs maintains 100% test coverage under the latest Python stable release (currently
-Python 3.9). Both unit tests and docstring doctests are included when computing
-coverage. Running ``pytest -v`` will automatically run the test suite with coverage
-and produce a coverage report. This should be 100% before code can be accepted into the
-main code base.
+NumCodecs maintains 100% test coverage under the latest Python stable release.
+Both unit tests and docstring doctests are included when computing coverage. Running
+``pytest -v`` will automatically run the test suite with coverage and produce a
+coverage report. This should be 100% before code can be accepted into the main
+code base.
 
 When submitting a pull request, coverage will also be collected across all supported
 Python versions via the Codecov service, and will be reported back within the pull
@@ -179,7 +178,7 @@ Documentation
 Docstrings for user-facing classes and functions should follow the `numpydoc
 <https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard>`_ standard,
 including sections for Parameters and Examples. All examples will be run as doctests
-under Python 3.9.
+under a stable version of Python.
 
 NumCodecs uses Sphinx for documentation, hosted on readthedocs.org. Documentation is
 written in the RestructuredText markup language (.rst files) in the ``docs`` folder.
@@ -207,8 +206,8 @@ Pull requests submitted by an external contributor should be reviewed and approv
 one core developers before being merged. Ideally, pull requests submitted by a core developer
 should be reviewed and approved by at least one other core developers before being merged.
 
-Pull requests should not be merged until all CI checks have passed (Travis, AppVeyor,
-Codecov) against code that has had the latest main merged in.
+Pull requests should not be merged until all CI checks have passed (GitHub Actions,
+CodeCov) against code that has had the latest main merged in.
 
 Compatibility and versioning policies
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From 3c2b0bed9d9855245783960c4622067c57faa845 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 24 Apr 2024 23:25:22 +0530
Subject: [PATCH 03/43] Add patch to disable threading and multiprocessing

---
 ...disable-multiprocessing-and-pthreads.patch | 47 +++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100644 tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch

diff --git a/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch b/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
new file mode 100644
index 00000000..8266e3bb
--- /dev/null
+++ b/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
@@ -0,0 +1,47 @@
+This patch disables multiprocessing and pthread for blosc. This file
+is attributed to the Pyodide developers and can be viewed at the upstream
+Pyodide repository at the following link:
+
+https://github.com/pyodide/pyodide/blob/d32e376013d8977b66c6aa828042b1fee8047aea/packages/numcodecs/patches/fixblosc.patch
+
+
+diff --git a/numcodecs/blosc.pyx b/numcodecs/blosc.pyx
+index 3bfdc00..c6521b7 100644
+--- a/numcodecs/blosc.pyx
++++ b/numcodecs/blosc.pyx
+@@ -77,6 +77,8 @@ AUTOBLOCKS = 0
+ # synchronization
+ try:
+     mutex = multiprocessing.Lock()
++except ModuleNotFoundError:
++    mutex = None
+ except OSError:
+     mutex = None
+ 
+diff --git a/c-blosc/blosc/blosc.h b/c-blosc/blosc/blosc.h
+index 40857d0..8a1e969 100644
+--- a/c-blosc/blosc/blosc.h
++++ b/c-blosc/blosc/blosc.h
+@@ -50,7 +50,7 @@ extern "C" {
+   ((INT_MAX - BLOSC_MAX_TYPESIZE * sizeof(int32_t)) / 3)
+ 
+ /* The maximum number of threads (for some static arrays) */
+-#define BLOSC_MAX_THREADS 256
++#define BLOSC_MAX_THREADS 1
+ 
+ /* Codes for shuffling (see blosc_compress) */
+ #define BLOSC_NOSHUFFLE   0  /* no shuffle */
+
+ diff --git a/c-blosc/blosc/blosc.c b/c-blosc/blosc/blosc.c
+index a5a5bd5..2a7797c 100644
+--- a/c-blosc/blosc/blosc.c
++++ b/c-blosc/blosc/blosc.c
+@@ -2236,6 +2236,7 @@ void blosc_atfork_child(void) {
+ 
+ void blosc_init(void)
+ {
++  g_initlib = 1;
+   /* Return if we are already initialized */
+   if (g_initlib) return;
+ 
+

From a8c956fd67687c481d3fdc7a6395ef3b5bee9e5a Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 24 Apr 2024 23:26:09 +0530
Subject: [PATCH 04/43] Add patch to embed missing POSIX syscall headers

---
 .../0002-add-missing-unistd-headers.patch     | 49 +++++++++++++++++++
 1 file changed, 49 insertions(+)
 create mode 100644 tools/ci/patches/0002-add-missing-unistd-headers.patch

diff --git a/tools/ci/patches/0002-add-missing-unistd-headers.patch b/tools/ci/patches/0002-add-missing-unistd-headers.patch
new file mode 100644
index 00000000..64f971ee
--- /dev/null
+++ b/tools/ci/patches/0002-add-missing-unistd-headers.patch
@@ -0,0 +1,49 @@
+This patch adds missing headers in vendored zlib as done in numcodecs.js. This file
+is attributed to the Pyodide developers and can be viewed at the upstream
+Pyodide repository at the following link;
+
+https://github.com/pyodide/pyodide/blob/d32e376013d8977b66c6aa828042b1fee8047aea/packages/numcodecs/patches/fixzlib.patch
+
+
+diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c b/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
+index fae202e..80606a6 100644
+--- a/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
++++ b/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
+@@ -2,7 +2,7 @@
+  * Copyright (C) 2004, 2010, 2011, 2012, 2013 Mark Adler
+  * For conditions of distribution and use, see copyright notice in zlib.h
+  */
+-
++#include <unistd.h>
+ #include "gzguts.h"
+ 
+ #if defined(_WIN32) && !defined(__BORLANDC__)
+
+diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzread.c b/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
+index bf4538e..afe6acd 100644
+--- a/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
++++ b/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
+@@ -2,7 +2,7 @@
+  * Copyright (C) 2004, 2005, 2010, 2011, 2012, 2013 Mark Adler
+  * For conditions of distribution and use, see copyright notice in zlib.h
+  */
+-
++#include <unistd.h>
+ #include "gzguts.h"
+ 
+ /* Local functions */
+
+
+diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c b/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
+index aa767fb..a87676f 100644
+--- a/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
++++ b/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
+@@ -2,7 +2,7 @@
+  * Copyright (C) 2004, 2005, 2010, 2011, 2012, 2013 Mark Adler
+  * For conditions of distribution and use, see copyright notice in zlib.h
+  */
+-
++#include <unistd.h>
+ #include "gzguts.h"
+ 
+ /* Local functions */
\ No newline at end of file

From 876974344d6b9909c6d3d756d8cd8e61a77f83e3 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 25 Apr 2024 00:02:23 +0530
Subject: [PATCH 05/43] Add initial workflow to run + debug Pyodide builds

---
 .github/workflows/ci-emscripten.yaml | 78 ++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
 create mode 100644 .github/workflows/ci-emscripten.yaml

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
new file mode 100644
index 00000000..bf5cc42e
--- /dev/null
+++ b/.github/workflows/ci-emscripten.yaml
@@ -0,0 +1,78 @@
+# Attributed to NumPy https://github.com/numpy/numpy/pull/25894
+# https://github.com/numpy/numpy/blob/d2d2c25fa81b47810f5cbd85ea6485eb3a3ffec3/.github/workflows/emscripten.yml
+#
+
+name: Pyodide CI
+
+on:
+  # TODO: refine after this is ready to merge
+  [push, pull_request, workflow_dispatch]
+
+env:
+  FORCE_COLOR: 3
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read  # to fetch code (actions/checkout)
+
+jobs:
+  build-wasm-emscripten:
+    name: Build numcodecs distribution for Pyodide
+    runs-on: ubuntu-22.04
+    # To enable this workflow on a fork, comment out:
+    # FIXME: uncomment after this is ready to merge
+    # if: github.repository == 'zarr-developers/numcodecs'
+    env:
+      PYODIDE_VERSION: 0.25.1
+      # PYTHON_VERSION and EMSCRIPTEN_VERSION are determined by PYODIDE_VERSION.
+      # The appropriate versions can be found in the Pyodide repodata.json
+      # "info" field, or in Makefile.envs:
+      # https://github.com/pyodide/pyodide/blob/main/Makefile.envs#L2
+      PYTHON_VERSION: 3.11.3
+      EMSCRIPTEN_VERSION: 3.1.46
+      NODE_VERSION: 18
+    steps:
+      - name: Checkout source
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Set up Python ${{ env.PYTHON_VERSION }}
+        id: setup-python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Set up Emscripten toolchain
+        uses: mymindstorm/setup-emsdk@v14
+        with:
+          version: ${{ env.EMSCRIPTEN_VERSION }}
+          actions-cache-folder: emsdk-cache
+
+      - name: Apply necessary patch(es)
+        run: |
+          patch -p1 < tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
+          patch -p1 < tools/ci/patches/0002-add-missing-unistd-headers.patch
+
+      - name: Install pyodide-build
+        run: python -m pip install "pydantic<2" "pyodide-build==${{ env.PYODIDE_VERSION }}"
+
+      - name: Build numcodecs for Pyodide
+        run: |
+          pyodide build
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+
+      - name: Set up Pyodide virtual environment and test numcodecs for Pyodide
+        run: |
+          pyodide venv .venv-pyodide
+          source .venv-pyodide/bin/activate
+          python -m pip install dist/*.whl
+          python -m pip install pytest pytest-cov coverage
+          python -m pytest --pyargs numcodecs

From 28674c3684edc77fe34a1d9a05f44fbfd2ac0207 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 25 Apr 2024 00:17:35 +0530
Subject: [PATCH 06/43] Disable AVX2 and SSE2 instructions

---
 .github/workflows/ci-emscripten.yaml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index bf5cc42e..d78efe7f 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -10,6 +10,10 @@ on:
 
 env:
   FORCE_COLOR: 3
+  # Disable instructions: AVX2 and SSE2 because Emscripten-specific SIMD
+  # support has not been implemented yet
+  DISABLE_NUMCODECS_AVX2: 1
+  DISABLE_NUMCODECS_SSE2: 1
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}

From f6df4881e6d7d6c4937e313a70682819b6a5622b Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 25 Apr 2024 00:49:52 +0530
Subject: [PATCH 07/43] Use separate build and test jobs and share wheel

---
 .github/workflows/ci-emscripten.yaml | 57 +++++++++++++++++++++++-----
 1 file changed, 47 insertions(+), 10 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index d78efe7f..f3850195 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -14,6 +14,15 @@ env:
   # support has not been implemented yet
   DISABLE_NUMCODECS_AVX2: 1
   DISABLE_NUMCODECS_SSE2: 1
+  # Common environment variables for both build and test jobs
+  PYODIDE_VERSION: 0.25.1
+  # PYTHON_VERSION and EMSCRIPTEN_VERSION are determined by PYODIDE_VERSION.
+  # The appropriate versions can be found in the Pyodide repodata.json
+  # "info" field, or in Makefile.envs:
+  # https://github.com/pyodide/pyodide/blob/main/Makefile.envs#L2
+  PYTHON_VERSION: 3.11.3
+  EMSCRIPTEN_VERSION: 3.1.46
+  NODE_VERSION: 18
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
@@ -24,20 +33,11 @@ permissions:
 
 jobs:
   build-wasm-emscripten:
-    name: Build numcodecs distribution for Pyodide
+    name: Build numcodecs Pyodide distribution
     runs-on: ubuntu-22.04
     # To enable this workflow on a fork, comment out:
     # FIXME: uncomment after this is ready to merge
     # if: github.repository == 'zarr-developers/numcodecs'
-    env:
-      PYODIDE_VERSION: 0.25.1
-      # PYTHON_VERSION and EMSCRIPTEN_VERSION are determined by PYODIDE_VERSION.
-      # The appropriate versions can be found in the Pyodide repodata.json
-      # "info" field, or in Makefile.envs:
-      # https://github.com/pyodide/pyodide/blob/main/Makefile.envs#L2
-      PYTHON_VERSION: 3.11.3
-      EMSCRIPTEN_VERSION: 3.1.46
-      NODE_VERSION: 18
     steps:
       - name: Checkout source
         uses: actions/checkout@v4
@@ -73,6 +73,43 @@ jobs:
         with:
           node-version: ${{ env.NODE_VERSION }}
 
+      - name: Upload Pyodide wheel artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: numcodecs-pyodide-wheel
+          path: dist/*.whl
+
+  test-wasm-emscripten:
+    name: Test numcodecs Pyodide distribution
+    runs-on: ubuntu-22.04
+    needs: [build-wasm-emscripten]
+    steps:
+      - name: Set up Python ${{ env.PYTHON_VERSION }}
+        id: setup-python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Set up Emscripten toolchain
+        uses: mymindstorm/setup-emsdk@v14
+        with:
+          version: ${{ env.EMSCRIPTEN_VERSION }}
+          actions-cache-folder: emsdk-cache
+
+      - name: Install pyodide-build
+        run: python -m pip install "pydantic<2" "pyodide-build==${{ env.PYODIDE_VERSION }}"
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+
+      - name: Download Pyodide wheel artifact
+        uses: actions/download-artifact@v4
+        with:
+          path: dist
+          merge-multiple: true
+
       - name: Set up Pyodide virtual environment and test numcodecs for Pyodide
         run: |
           pyodide venv .venv-pyodide

From d678be826f99ef3dd8e08acc3f190575738f8a0f Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 25 Apr 2024 00:56:35 +0530
Subject: [PATCH 08/43] Add patch that allows loading `.npy` files

---
 .../ci/patches/0003-fix-npy-file-access.patch | 172 ++++++++++++++++++
 1 file changed, 172 insertions(+)
 create mode 100644 tools/ci/patches/0003-fix-npy-file-access.patch

diff --git a/tools/ci/patches/0003-fix-npy-file-access.patch b/tools/ci/patches/0003-fix-npy-file-access.patch
new file mode 100644
index 00000000..920371fd
--- /dev/null
+++ b/tools/ci/patches/0003-fix-npy-file-access.patch
@@ -0,0 +1,172 @@
+From 135baa3d4cc211c0145d2e1df116e39f60df4f91 Mon Sep 17 00:00:00 2001
+From: Hood Chatham <roberthoodchatham@gmail.com>
+Date: Thu, 22 Feb 2024 21:47:13 -0800
+Subject: [PATCH] Fix dup in nodefs by refcounting nodefs file descriptors
+ (#21399)
+
+This patch is sourced from the upstream patch in the Pyodide project:
+https://github.com/pyodide/pyodide/blob/163a46e3ca5128a5c25b79bb6b4db76b7f5d9b30/emsdk/patches/0001-Fix-dup-in-nodefs-by-refcounting-nodefs-file-descrip.patch
+It fixes a bug in the NodeFS filesystem for Emscripten/emsdk where
+file descriptors were not being properly refcounted, causing them to
+be closed when they were duplicated. This patch fixes file system
+access for NumPy when loading data from .npy file formats.
+
+---
+ src/library_fs.js         |  5 +++++
+ src/library_nodefs.js     |  6 +++++-
+ src/library_syscall.js    |  6 +++---
+ test/fs/test_nodefs_dup.c | 45 +++++++++++++++++++++++++++++++++++++++
+ test/test_core.py         |  8 +++++++
+ 5 files changed, 66 insertions(+), 4 deletions(-)
+ create mode 100644 test/fs/test_nodefs_dup.c
+
+diff --git a/src/library_fs.js b/src/library_fs.js
+index f5d16b86c..379e65268 100644
+--- a/src/library_fs.js
++++ b/src/library_fs.js
+@@ -471,6 +471,11 @@ FS.staticInit();` +
+     closeStream(fd) {
+       FS.streams[fd] = null;
+     },
++    dupStream(origStream, fd = -1) {
++      var stream = FS.createStream(origStream, fd);
++      stream.stream_ops?.dup?.(stream);
++      return stream;
++    },
+
+     //
+     // devices
+diff --git a/src/library_nodefs.js b/src/library_nodefs.js
+index 81864ffcc..ace50458c 100644
+--- a/src/library_nodefs.js
++++ b/src/library_nodefs.js
+@@ -251,6 +251,7 @@ addToLibrary({
+         var path = NODEFS.realPath(stream.node);
+         try {
+           if (FS.isFile(stream.node.mode)) {
++            stream.shared.refcount = 1;
+             stream.nfd = fs.openSync(path, NODEFS.flagsForNode(stream.flags));
+           }
+         } catch (e) {
+@@ -260,7 +261,7 @@ addToLibrary({
+       },
+       close(stream) {
+         try {
+-          if (FS.isFile(stream.node.mode) && stream.nfd) {
++          if (FS.isFile(stream.node.mode) && stream.nfd && --stream.shared.refcount === 0) {
+             fs.closeSync(stream.nfd);
+           }
+         } catch (e) {
+@@ -268,6 +269,9 @@ addToLibrary({
+           throw new FS.ErrnoError(NODEFS.convertNodeCode(e));
+         }
+       },
++      dup(stream) {
++        stream.shared.refcount++;
++      },
+       read(stream, buffer, offset, length, position) {
+         // Node.js < 6 compatibility: node errors on 0 length reads
+         if (length === 0) return 0;
+diff --git a/src/library_syscall.js b/src/library_syscall.js
+index b078bd71c..69ea6f8b2 100644
+--- a/src/library_syscall.js
++++ b/src/library_syscall.js
+@@ -183,7 +183,7 @@ var SyscallsLibrary = {
+   },
+   __syscall_dup: (fd) => {
+     var old = SYSCALLS.getStreamFromFD(fd);
+-    return FS.createStream(old).fd;
++    return FS.dupStream(old).fd;
+   },
+   __syscall_pipe__deps: ['$PIPEFS'],
+   __syscall_pipe: (fdPtr) => {
+@@ -760,7 +760,7 @@ var SyscallsLibrary = {
+           arg++;
+         }
+         var newStream;
+-        newStream = FS.createStream(stream, arg);
++        newStream = FS.dupStream(stream, arg);
+         return newStream.fd;
+       }
+       case {{{ cDefs.F_GETFD }}}:
+@@ -1007,7 +1007,7 @@ var SyscallsLibrary = {
+     if (old.fd === newfd) return -{{{ cDefs.EINVAL }}};
+     var existing = FS.getStream(newfd);
+     if (existing) FS.close(existing);
+-    return FS.createStream(old, newfd).fd;
++    return FS.dupStream(old, newfd).fd;
+   },
+ };
+
+diff --git a/test/fs/test_nodefs_dup.c b/test/fs/test_nodefs_dup.c
+new file mode 100644
+index 000000000..abf34935b
+--- /dev/null
++++ b/test/fs/test_nodefs_dup.c
+@@ -0,0 +1,45 @@
++/*
++ * Copyright 2018 The Emscripten Authors.  All rights reserved.
++ * Emscripten is available under two separate licenses, the MIT license and the
++ * University of Illinois/NCSA Open Source License.  Both these licenses can be
++ * found in the LICENSE file.
++ */
++
++#include <assert.h>
++#include <emscripten.h>
++#include <fcntl.h>
++#include <stdio.h>
++#include <unistd.h>
++
++#ifdef NODERAWFS
++#define CWD ""
++#else
++#define CWD "/working/"
++#endif
++
++int main(void)
++{
++    EM_ASM(
++#ifdef NODERAWFS
++        FS.close(FS.open('test.txt', 'w'));
++#else
++        FS.mkdir('/working');
++        FS.mount(NODEFS, {root: '.'}, '/working');
++        FS.close(FS.open('/working/test.txt', 'w'));
++#endif
++    );
++    int fd1 = open(CWD "test.txt", O_WRONLY);
++    int fd2 = dup(fd1);
++    int fd3 = fcntl(fd1, F_DUPFD_CLOEXEC, 0);
++
++    assert(fd1 == 3);
++    assert(fd2 == 4);
++    assert(fd3 == 5);
++    assert(close(fd1) == 0);
++    assert(write(fd2, "abcdef", 6) == 6);
++    assert(close(fd2) == 0);
++    assert(write(fd3, "ghijkl", 6) == 6);
++    assert(close(fd3) == 0);
++    printf("success\n");
++    return 0;
++}
+diff --git a/test/test_core.py b/test/test_core.py
+index 3f21eb5ef..f304f1366 100644
+--- a/test/test_core.py
++++ b/test/test_core.py
+@@ -5745,6 +5745,14 @@ Module = {
+     self.emcc_args += ['-lnodefs.js']
+     self.do_runf('fs/test_nodefs_cloexec.c', 'success')
+
++  @also_with_noderawfs
++  @requires_node
++  def test_fs_nodefs_dup(self):
++    if self.get_setting('WASMFS'):
++      self.set_setting('FORCE_FILESYSTEM')
++    self.emcc_args += ['-lnodefs.js']
++    self.do_runf('fs/test_nodefs_dup.c', 'success')
++
+   @requires_node
+   def test_fs_nodefs_home(self):
+     self.set_setting('FORCE_FILESYSTEM')
+--
+2.34.1

From 6ce41275370c0c34b8d95d750489492a0b333655 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 25 Apr 2024 00:56:48 +0530
Subject: [PATCH 09/43] Apply patch for Emscripten to load `.npy` files

---
 .github/workflows/ci-emscripten.yaml | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index f3850195..e3e611af 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -60,6 +60,7 @@ jobs:
         run: |
           patch -p1 < tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
           patch -p1 < tools/ci/patches/0002-add-missing-unistd-headers.patch
+          patch -d $EMSDK/upstream/emscripten/ -p1 < tools/ci/patches/0003-fix-npy-file-access.patch
 
       - name: Install pyodide-build
         run: python -m pip install "pydantic<2" "pyodide-build==${{ env.PYODIDE_VERSION }}"
@@ -90,11 +91,11 @@ jobs:
         with:
           python-version: ${{ env.PYTHON_VERSION }}
 
-      - name: Set up Emscripten toolchain
-        uses: mymindstorm/setup-emsdk@v14
-        with:
-          version: ${{ env.EMSCRIPTEN_VERSION }}
-          actions-cache-folder: emsdk-cache
+      # - name: Set up Emscripten toolchain
+      #   uses: mymindstorm/setup-emsdk@v14
+      #   with:
+      #     version: ${{ env.EMSCRIPTEN_VERSION }}
+      #     actions-cache-folder: emsdk-cache
 
       - name: Install pyodide-build
         run: python -m pip install "pydantic<2" "pyodide-build==${{ env.PYODIDE_VERSION }}"

From 199928d28699d93490434dc244d6758f05ce2401 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 25 Apr 2024 01:18:53 +0530
Subject: [PATCH 10/43] Add a constant to check if platform is WASM

---
 numcodecs/tests/common.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/numcodecs/tests/common.py b/numcodecs/tests/common.py
index beff0f14..96162c4b 100644
--- a/numcodecs/tests/common.py
+++ b/numcodecs/tests/common.py
@@ -2,6 +2,8 @@
 import json as _json
 import os
 from glob import glob
+import platform
+import sys
 
 
 import numpy as np
@@ -20,6 +22,8 @@
              'こんにちは世界', '世界，你好！', 'Helló, világ!', 'Zdravo svete!',
              'เฮลโลเวิลด์']
 
+is_wasm = (sys.platform == 'emscripten') or (platform.machine() in ['wasm32', 'wasm64'])
+
 
 def compare_arrays(arr, res, precision=None):
 

From 1f5e7b7fce347d6fe5cf668803bd7b1644b5e7c0 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 25 Apr 2024 01:19:25 +0530
Subject: [PATCH 11/43] Use `is_wasm` to disable multiprocessing tests

---
 numcodecs/tests/test_blosc.py   | 7 ++++++-
 numcodecs/tests/test_shuffle.py | 7 ++++++-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/numcodecs/tests/test_blosc.py b/numcodecs/tests/test_blosc.py
index 618894b3..c4261ef0 100644
--- a/numcodecs/tests/test_blosc.py
+++ b/numcodecs/tests/test_blosc.py
@@ -15,7 +15,8 @@
     )
 
 
-from numcodecs.tests.common import (check_encode_decode,
+from numcodecs.tests.common import (is_wasm,
+                                    check_encode_decode,
                                     check_encode_decode_partial,
                                     check_config,
                                     check_backwards_compatibility,
@@ -225,6 +226,10 @@ def _decode_worker(enc):
     return data
 
 
+@pytest.mark.skipif(
+    is_wasm,
+    reason="WASM/Pyodide does not support multiprocessing"
+)
 @pytest.mark.parametrize('pool', (Pool, ThreadPool))
 def test_multiprocessing(use_threads, pool):
     data = np.arange(1000000)
diff --git a/numcodecs/tests/test_shuffle.py b/numcodecs/tests/test_shuffle.py
index 6c19ca1f..939f92c9 100644
--- a/numcodecs/tests/test_shuffle.py
+++ b/numcodecs/tests/test_shuffle.py
@@ -14,7 +14,8 @@
     )
 
 
-from numcodecs.tests.common import (check_encode_decode,
+from numcodecs.tests.common import (is_wasm,
+                                    check_encode_decode,
                                     check_config,
                                     check_backwards_compatibility)
 
@@ -92,6 +93,10 @@ def _decode_worker(enc):
     return data
 
 
+@pytest.mark.skipif(
+    is_wasm,
+    reason="WASM/Pyodide does not support multiprocessing"
+)
 @pytest.mark.parametrize('pool', (Pool, ThreadPool))
 def test_multiprocessing(pool):
     data = np.arange(1000000)

From da909f82265ce7acc733832969a9e4955f373636 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 25 Apr 2024 01:50:31 +0530
Subject: [PATCH 12/43] Apply `.npy` files patch again when running tests

---
 .github/workflows/ci-emscripten.yaml | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index e3e611af..22962cc0 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -91,11 +91,22 @@ jobs:
         with:
           python-version: ${{ env.PYTHON_VERSION }}
 
-      # - name: Set up Emscripten toolchain
-      #   uses: mymindstorm/setup-emsdk@v14
-      #   with:
-      #     version: ${{ env.EMSCRIPTEN_VERSION }}
-      #     actions-cache-folder: emsdk-cache
+      - name: Checkout necessary patch(es)
+        uses: actions/checkout@v4
+        with:
+          # Check out a single file
+          sparse-checkout: tools/ci/patches/0003-fix-npy-file-access.patch
+          sparse-checkout-cone-mode: false
+
+      - name: Set up Emscripten toolchain
+        uses: mymindstorm/setup-emsdk@v14
+        with:
+          version: ${{ env.EMSCRIPTEN_VERSION }}
+          actions-cache-folder: emsdk-cache
+
+      - name: Apply necessary patch(es)
+        # We just need the third patch to fix the npy file access this time
+        run: patch -d $EMSDK/upstream/emscripten/ -p1 < tools/ci/patches/0003-fix-npy-file-access.patch
 
       - name: Install pyodide-build
         run: python -m pip install "pydantic<2" "pyodide-build==${{ env.PYODIDE_VERSION }}"

From a54c555fd995d1bcfa820a639bb6b1092345eb81 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 22 May 2024 01:26:14 +0530
Subject: [PATCH 13/43] Remove patch that is unneeded at build time

---
 .github/workflows/ci-emscripten.yaml | 2 --
 1 file changed, 2 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index 22962cc0..dc87fa0f 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -60,7 +60,6 @@ jobs:
         run: |
           patch -p1 < tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
           patch -p1 < tools/ci/patches/0002-add-missing-unistd-headers.patch
-          patch -d $EMSDK/upstream/emscripten/ -p1 < tools/ci/patches/0003-fix-npy-file-access.patch
 
       - name: Install pyodide-build
         run: python -m pip install "pydantic<2" "pyodide-build==${{ env.PYODIDE_VERSION }}"
@@ -94,7 +93,6 @@ jobs:
       - name: Checkout necessary patch(es)
         uses: actions/checkout@v4
         with:
-          # Check out a single file
           sparse-checkout: tools/ci/patches/0003-fix-npy-file-access.patch
           sparse-checkout-cone-mode: false
 

From a7446c8ca99495f6dd3e3da3e572fddd6e9b656c Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 19:46:26 +0530
Subject: [PATCH 14/43] Delete patch that's not needed now

---
 .../ci/patches/0003-fix-npy-file-access.patch | 172 ------------------
 1 file changed, 172 deletions(-)
 delete mode 100644 tools/ci/patches/0003-fix-npy-file-access.patch

diff --git a/tools/ci/patches/0003-fix-npy-file-access.patch b/tools/ci/patches/0003-fix-npy-file-access.patch
deleted file mode 100644
index 920371fd..00000000
--- a/tools/ci/patches/0003-fix-npy-file-access.patch
+++ /dev/null
@@ -1,172 +0,0 @@
-From 135baa3d4cc211c0145d2e1df116e39f60df4f91 Mon Sep 17 00:00:00 2001
-From: Hood Chatham <roberthoodchatham@gmail.com>
-Date: Thu, 22 Feb 2024 21:47:13 -0800
-Subject: [PATCH] Fix dup in nodefs by refcounting nodefs file descriptors
- (#21399)
-
-This patch is sourced from the upstream patch in the Pyodide project:
-https://github.com/pyodide/pyodide/blob/163a46e3ca5128a5c25b79bb6b4db76b7f5d9b30/emsdk/patches/0001-Fix-dup-in-nodefs-by-refcounting-nodefs-file-descrip.patch
-It fixes a bug in the NodeFS filesystem for Emscripten/emsdk where
-file descriptors were not being properly refcounted, causing them to
-be closed when they were duplicated. This patch fixes file system
-access for NumPy when loading data from .npy file formats.
-
----
- src/library_fs.js         |  5 +++++
- src/library_nodefs.js     |  6 +++++-
- src/library_syscall.js    |  6 +++---
- test/fs/test_nodefs_dup.c | 45 +++++++++++++++++++++++++++++++++++++++
- test/test_core.py         |  8 +++++++
- 5 files changed, 66 insertions(+), 4 deletions(-)
- create mode 100644 test/fs/test_nodefs_dup.c
-
-diff --git a/src/library_fs.js b/src/library_fs.js
-index f5d16b86c..379e65268 100644
---- a/src/library_fs.js
-+++ b/src/library_fs.js
-@@ -471,6 +471,11 @@ FS.staticInit();` +
-     closeStream(fd) {
-       FS.streams[fd] = null;
-     },
-+    dupStream(origStream, fd = -1) {
-+      var stream = FS.createStream(origStream, fd);
-+      stream.stream_ops?.dup?.(stream);
-+      return stream;
-+    },
-
-     //
-     // devices
-diff --git a/src/library_nodefs.js b/src/library_nodefs.js
-index 81864ffcc..ace50458c 100644
---- a/src/library_nodefs.js
-+++ b/src/library_nodefs.js
-@@ -251,6 +251,7 @@ addToLibrary({
-         var path = NODEFS.realPath(stream.node);
-         try {
-           if (FS.isFile(stream.node.mode)) {
-+            stream.shared.refcount = 1;
-             stream.nfd = fs.openSync(path, NODEFS.flagsForNode(stream.flags));
-           }
-         } catch (e) {
-@@ -260,7 +261,7 @@ addToLibrary({
-       },
-       close(stream) {
-         try {
--          if (FS.isFile(stream.node.mode) && stream.nfd) {
-+          if (FS.isFile(stream.node.mode) && stream.nfd && --stream.shared.refcount === 0) {
-             fs.closeSync(stream.nfd);
-           }
-         } catch (e) {
-@@ -268,6 +269,9 @@ addToLibrary({
-           throw new FS.ErrnoError(NODEFS.convertNodeCode(e));
-         }
-       },
-+      dup(stream) {
-+        stream.shared.refcount++;
-+      },
-       read(stream, buffer, offset, length, position) {
-         // Node.js < 6 compatibility: node errors on 0 length reads
-         if (length === 0) return 0;
-diff --git a/src/library_syscall.js b/src/library_syscall.js
-index b078bd71c..69ea6f8b2 100644
---- a/src/library_syscall.js
-+++ b/src/library_syscall.js
-@@ -183,7 +183,7 @@ var SyscallsLibrary = {
-   },
-   __syscall_dup: (fd) => {
-     var old = SYSCALLS.getStreamFromFD(fd);
--    return FS.createStream(old).fd;
-+    return FS.dupStream(old).fd;
-   },
-   __syscall_pipe__deps: ['$PIPEFS'],
-   __syscall_pipe: (fdPtr) => {
-@@ -760,7 +760,7 @@ var SyscallsLibrary = {
-           arg++;
-         }
-         var newStream;
--        newStream = FS.createStream(stream, arg);
-+        newStream = FS.dupStream(stream, arg);
-         return newStream.fd;
-       }
-       case {{{ cDefs.F_GETFD }}}:
-@@ -1007,7 +1007,7 @@ var SyscallsLibrary = {
-     if (old.fd === newfd) return -{{{ cDefs.EINVAL }}};
-     var existing = FS.getStream(newfd);
-     if (existing) FS.close(existing);
--    return FS.createStream(old, newfd).fd;
-+    return FS.dupStream(old, newfd).fd;
-   },
- };
-
-diff --git a/test/fs/test_nodefs_dup.c b/test/fs/test_nodefs_dup.c
-new file mode 100644
-index 000000000..abf34935b
---- /dev/null
-+++ b/test/fs/test_nodefs_dup.c
-@@ -0,0 +1,45 @@
-+/*
-+ * Copyright 2018 The Emscripten Authors.  All rights reserved.
-+ * Emscripten is available under two separate licenses, the MIT license and the
-+ * University of Illinois/NCSA Open Source License.  Both these licenses can be
-+ * found in the LICENSE file.
-+ */
-+
-+#include <assert.h>
-+#include <emscripten.h>
-+#include <fcntl.h>
-+#include <stdio.h>
-+#include <unistd.h>
-+
-+#ifdef NODERAWFS
-+#define CWD ""
-+#else
-+#define CWD "/working/"
-+#endif
-+
-+int main(void)
-+{
-+    EM_ASM(
-+#ifdef NODERAWFS
-+        FS.close(FS.open('test.txt', 'w'));
-+#else
-+        FS.mkdir('/working');
-+        FS.mount(NODEFS, {root: '.'}, '/working');
-+        FS.close(FS.open('/working/test.txt', 'w'));
-+#endif
-+    );
-+    int fd1 = open(CWD "test.txt", O_WRONLY);
-+    int fd2 = dup(fd1);
-+    int fd3 = fcntl(fd1, F_DUPFD_CLOEXEC, 0);
-+
-+    assert(fd1 == 3);
-+    assert(fd2 == 4);
-+    assert(fd3 == 5);
-+    assert(close(fd1) == 0);
-+    assert(write(fd2, "abcdef", 6) == 6);
-+    assert(close(fd2) == 0);
-+    assert(write(fd3, "ghijkl", 6) == 6);
-+    assert(close(fd3) == 0);
-+    printf("success\n");
-+    return 0;
-+}
-diff --git a/test/test_core.py b/test/test_core.py
-index 3f21eb5ef..f304f1366 100644
---- a/test/test_core.py
-+++ b/test/test_core.py
-@@ -5745,6 +5745,14 @@ Module = {
-     self.emcc_args += ['-lnodefs.js']
-     self.do_runf('fs/test_nodefs_cloexec.c', 'success')
-
-+  @also_with_noderawfs
-+  @requires_node
-+  def test_fs_nodefs_dup(self):
-+    if self.get_setting('WASMFS'):
-+      self.set_setting('FORCE_FILESYSTEM')
-+    self.emcc_args += ['-lnodefs.js']
-+    self.do_runf('fs/test_nodefs_dup.c', 'success')
-+
-   @requires_node
-   def test_fs_nodefs_home(self):
-     self.set_setting('FORCE_FILESYSTEM')
---
-2.34.1

From 9271054da86297221c83c07aa56b485b686ebb58 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 19:46:43 +0530
Subject: [PATCH 15/43] Ignore Pyodide cross-build environment files

---
 .gitignore | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.gitignore b/.gitignore
index c7d09ec5..80ce22b8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -105,3 +105,6 @@ numcodecs/version.py
 
 # Cython generated
 numcodecs/*.c
+
+# Pyodide builds
+/.pyodide-xbuildenv-*

From 764f34e5c0fd1c31a5316af760e3fc1848e28410 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 19:47:16 +0530
Subject: [PATCH 16/43] Revamp workflow and clean up changes

---
 .github/workflows/ci-emscripten.yaml | 72 +++++++---------------------
 1 file changed, 17 insertions(+), 55 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index dc87fa0f..f2a5a4ab 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -15,14 +15,14 @@ env:
   DISABLE_NUMCODECS_AVX2: 1
   DISABLE_NUMCODECS_SSE2: 1
   # Common environment variables for both build and test jobs
-  PYODIDE_VERSION: 0.25.1
+  PYODIDE_VERSION: 0.27.0a2
   # PYTHON_VERSION and EMSCRIPTEN_VERSION are determined by PYODIDE_VERSION.
   # The appropriate versions can be found in the Pyodide repodata.json
   # "info" field, or in Makefile.envs:
   # https://github.com/pyodide/pyodide/blob/main/Makefile.envs#L2
-  PYTHON_VERSION: 3.11.3
-  EMSCRIPTEN_VERSION: 3.1.46
-  NODE_VERSION: 18
+  PYTHON_VERSION: 3.12.1
+  EMSCRIPTEN_VERSION: 3.1.58
+  NODE_VERSION: 20
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
@@ -62,68 +62,30 @@ jobs:
           patch -p1 < tools/ci/patches/0002-add-missing-unistd-headers.patch
 
       - name: Install pyodide-build
-        run: python -m pip install "pydantic<2" "pyodide-build==${{ env.PYODIDE_VERSION }}"
+        run: python -m pip install pyodide-build
 
-      - name: Build numcodecs for Pyodide
-        run: |
-          pyodide build
-
-      - name: Set up Node.js
-        uses: actions/setup-node@v4
-        with:
-          node-version: ${{ env.NODE_VERSION }}
-
-      - name: Upload Pyodide wheel artifact
-        uses: actions/upload-artifact@v4
-        with:
-          name: numcodecs-pyodide-wheel
-          path: dist/*.whl
-
-  test-wasm-emscripten:
-    name: Test numcodecs Pyodide distribution
-    runs-on: ubuntu-22.04
-    needs: [build-wasm-emscripten]
-    steps:
-      - name: Set up Python ${{ env.PYTHON_VERSION }}
-        id: setup-python
-        uses: actions/setup-python@v5
-        with:
-          python-version: ${{ env.PYTHON_VERSION }}
-
-      - name: Checkout necessary patch(es)
-        uses: actions/checkout@v4
-        with:
-          sparse-checkout: tools/ci/patches/0003-fix-npy-file-access.patch
-          sparse-checkout-cone-mode: false
-
-      - name: Set up Emscripten toolchain
-        uses: mymindstorm/setup-emsdk@v14
-        with:
-          version: ${{ env.EMSCRIPTEN_VERSION }}
-          actions-cache-folder: emsdk-cache
-
-      - name: Apply necessary patch(es)
-        # We just need the third patch to fix the npy file access this time
-        run: patch -d $EMSDK/upstream/emscripten/ -p1 < tools/ci/patches/0003-fix-npy-file-access.patch
-
-      - name: Install pyodide-build
-        run: python -m pip install "pydantic<2" "pyodide-build==${{ env.PYODIDE_VERSION }}"
+      - name: Build numcodecs for Pyodide/WASM
+        run: pyodide build
 
       - name: Set up Node.js
         uses: actions/setup-node@v4
         with:
           node-version: ${{ env.NODE_VERSION }}
 
-      - name: Download Pyodide wheel artifact
-        uses: actions/download-artifact@v4
-        with:
-          path: dist
-          merge-multiple: true
-
       - name: Set up Pyodide virtual environment and test numcodecs for Pyodide
         run: |
+          # Pin to a specific version of Pyodide to ensure reliability
+          pyodide xbuildenv install ${{ env.PYODIDE_VERSION }}
+
+          # Set up Pyodide virtual environment and activate it
           pyodide venv .venv-pyodide
           source .venv-pyodide/bin/activate
+
+          # Install the built numcodecs WASM package
           python -m pip install dist/*.whl
           python -m pip install pytest pytest-cov coverage
+
+          # Change into a different directory before running tests to avoid
+          # the test runner picking up the local numcodecs package
+          cd docs
           python -m pytest --pyargs numcodecs

From d41f75fa893f6db23b676d37efb217e4575e90fa Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 19:50:50 +0530
Subject: [PATCH 17/43] Fix linter errors and style failures

---
 numcodecs/tests/common.py                              |  2 +-
 numcodecs/tests/test_blosc.py                          |  9 +++------
 numcodecs/tests/test_shuffle.py                        |  9 +++------
 .../0001-disable-multiprocessing-and-pthreads.patch    | 10 +++++-----
 tools/ci/patches/0002-add-missing-unistd-headers.patch |  6 +++---
 5 files changed, 15 insertions(+), 21 deletions(-)

diff --git a/numcodecs/tests/common.py b/numcodecs/tests/common.py
index b5570554..22df957e 100644
--- a/numcodecs/tests/common.py
+++ b/numcodecs/tests/common.py
@@ -1,9 +1,9 @@
 import array
 import json as _json
 import os
-from glob import glob
 import platform
 import sys
+from glob import glob
 
 import numpy as np
 import pytest
diff --git a/numcodecs/tests/test_blosc.py b/numcodecs/tests/test_blosc.py
index 8969ed15..4a5f6177 100644
--- a/numcodecs/tests/test_blosc.py
+++ b/numcodecs/tests/test_blosc.py
@@ -12,7 +12,6 @@
 
 
 from numcodecs.tests.common import (
-    is_wasm,
     check_backwards_compatibility,
     check_config,
     check_encode_decode,
@@ -20,6 +19,7 @@
     check_err_decode_object_buffer,
     check_err_encode_object_buffer,
     check_max_buffer_size,
+    is_wasm,
 )
 
 codecs = [
@@ -226,11 +226,8 @@ def _decode_worker(enc):
     return data
 
 
-@pytest.mark.skipif(
-    is_wasm,
-    reason="WASM/Pyodide does not support multiprocessing"
-)
-@pytest.mark.parametrize('pool', (Pool, ThreadPool))
+@pytest.mark.skipif(is_wasm, reason="WASM/Pyodide does not support multiprocessing")
+@pytest.mark.parametrize('pool', [Pool, ThreadPool])
 def test_multiprocessing(use_threads, pool):
     data = np.arange(1000000)
     enc = _encode_worker(data)
diff --git a/numcodecs/tests/test_shuffle.py b/numcodecs/tests/test_shuffle.py
index d644bd72..f97ff490 100644
--- a/numcodecs/tests/test_shuffle.py
+++ b/numcodecs/tests/test_shuffle.py
@@ -11,10 +11,10 @@
 
 
 from numcodecs.tests.common import (
-    is_wasm,
     check_backwards_compatibility,
     check_config,
     check_encode_decode,
+    is_wasm,
 )
 
 codecs = [
@@ -90,11 +90,8 @@ def _decode_worker(enc):
     return data
 
 
-@pytest.mark.skipif(
-    is_wasm,
-    reason="WASM/Pyodide does not support multiprocessing"
-)
-@pytest.mark.parametrize('pool', (Pool, ThreadPool))
+@pytest.mark.skipif(is_wasm, reason="WASM/Pyodide does not support multiprocessing")
+@pytest.mark.parametrize('pool', [Pool, ThreadPool])
 def test_multiprocessing(pool):
     data = np.arange(1000000)
     enc = _encode_worker(data)
diff --git a/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch b/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
index 8266e3bb..4fbc940e 100644
--- a/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
+++ b/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
@@ -17,18 +17,18 @@ index 3bfdc00..c6521b7 100644
 +    mutex = None
  except OSError:
      mutex = None
- 
+
 diff --git a/c-blosc/blosc/blosc.h b/c-blosc/blosc/blosc.h
 index 40857d0..8a1e969 100644
 --- a/c-blosc/blosc/blosc.h
 +++ b/c-blosc/blosc/blosc.h
 @@ -50,7 +50,7 @@ extern "C" {
    ((INT_MAX - BLOSC_MAX_TYPESIZE * sizeof(int32_t)) / 3)
- 
+
  /* The maximum number of threads (for some static arrays) */
 -#define BLOSC_MAX_THREADS 256
 +#define BLOSC_MAX_THREADS 1
- 
+
  /* Codes for shuffling (see blosc_compress) */
  #define BLOSC_NOSHUFFLE   0  /* no shuffle */
 
@@ -37,11 +37,11 @@ index a5a5bd5..2a7797c 100644
 --- a/c-blosc/blosc/blosc.c
 +++ b/c-blosc/blosc/blosc.c
 @@ -2236,6 +2236,7 @@ void blosc_atfork_child(void) {
- 
+
  void blosc_init(void)
  {
 +  g_initlib = 1;
    /* Return if we are already initialized */
    if (g_initlib) return;
- 
+
 
diff --git a/tools/ci/patches/0002-add-missing-unistd-headers.patch b/tools/ci/patches/0002-add-missing-unistd-headers.patch
index 64f971ee..428a63ae 100644
--- a/tools/ci/patches/0002-add-missing-unistd-headers.patch
+++ b/tools/ci/patches/0002-add-missing-unistd-headers.patch
@@ -16,7 +16,7 @@ index fae202e..80606a6 100644
 -
 +#include <unistd.h>
  #include "gzguts.h"
- 
+
  #if defined(_WIN32) && !defined(__BORLANDC__)
 
 diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzread.c b/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
@@ -30,7 +30,7 @@ index bf4538e..afe6acd 100644
 -
 +#include <unistd.h>
  #include "gzguts.h"
- 
+
  /* Local functions */
 
 
@@ -45,5 +45,5 @@ index aa767fb..a87676f 100644
 -
 +#include <unistd.h>
  #include "gzguts.h"
- 
+
  /* Local functions */
\ No newline at end of file

From ee39d1e5fd395be68c2c37a9c94c95efd5166a3b Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 20:43:10 +0530
Subject: [PATCH 18/43] Catch and disable `mutex` assignment

---
 numcodecs/blosc.pyx | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/numcodecs/blosc.pyx b/numcodecs/blosc.pyx
index 2c43dd04..bcc8acb2 100644
--- a/numcodecs/blosc.pyx
+++ b/numcodecs/blosc.pyx
@@ -87,6 +87,8 @@ def get_mutex():
             mutex = None
         except ImportError:
             mutex = None
+        except ModuleNotFoundError:
+            mutex = None
         _MUTEX = mutex
         _MUTEX_IS_INIT = True
     return _MUTEX

From 731c6b2caf4b3954b8850cc13eb2bba051aab728 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 20:43:25 +0530
Subject: [PATCH 19/43] Fix patch that disables multiprocessing, threading

---
 ...1-disable-multiprocessing-and-pthreads.patch | 17 ++---------------
 1 file changed, 2 insertions(+), 15 deletions(-)

diff --git a/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch b/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
index 4fbc940e..dcab2de0 100644
--- a/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
+++ b/tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
@@ -1,23 +1,10 @@
 This patch disables multiprocessing and pthread for blosc. This file
-is attributed to the Pyodide developers and can be viewed at the upstream
-Pyodide repository at the following link:
+is adapted from and attributed to the Pyodide developers and can be
+viewed at the upstream Pyodide repository at the following link:
 
 https://github.com/pyodide/pyodide/blob/d32e376013d8977b66c6aa828042b1fee8047aea/packages/numcodecs/patches/fixblosc.patch
 
 
-diff --git a/numcodecs/blosc.pyx b/numcodecs/blosc.pyx
-index 3bfdc00..c6521b7 100644
---- a/numcodecs/blosc.pyx
-+++ b/numcodecs/blosc.pyx
-@@ -77,6 +77,8 @@ AUTOBLOCKS = 0
- # synchronization
- try:
-     mutex = multiprocessing.Lock()
-+except ModuleNotFoundError:
-+    mutex = None
- except OSError:
-     mutex = None
-
 diff --git a/c-blosc/blosc/blosc.h b/c-blosc/blosc/blosc.h
 index 40857d0..8a1e969 100644
 --- a/c-blosc/blosc/blosc.h

From 0616dd5b490f410e74093cae4fec57ba4678aa17 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 20:56:01 +0530
Subject: [PATCH 20/43] Don't hardcode `zlib` paths in patch details

---
 .github/workflows/ci-emscripten.yaml          |  2 +-
 .../0002-add-missing-unistd-headers.patch     | 20 ++++++++++---------
 2 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index f2a5a4ab..17b1f1c6 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -59,7 +59,7 @@ jobs:
       - name: Apply necessary patch(es)
         run: |
           patch -p1 < tools/ci/patches/0001-disable-multiprocessing-and-pthreads.patch
-          patch -p1 < tools/ci/patches/0002-add-missing-unistd-headers.patch
+          patch -p1 < tools/ci/patches/0002-add-missing-unistd-headers.patch -d c-blosc/internal-complibs/zlib-*/
 
       - name: Install pyodide-build
         run: python -m pip install pyodide-build
diff --git a/tools/ci/patches/0002-add-missing-unistd-headers.patch b/tools/ci/patches/0002-add-missing-unistd-headers.patch
index 428a63ae..e7eb430e 100644
--- a/tools/ci/patches/0002-add-missing-unistd-headers.patch
+++ b/tools/ci/patches/0002-add-missing-unistd-headers.patch
@@ -4,11 +4,13 @@ Pyodide repository at the following link;
 
 https://github.com/pyodide/pyodide/blob/d32e376013d8977b66c6aa828042b1fee8047aea/packages/numcodecs/patches/fixzlib.patch
 
+This patch is applied in the c-blosc/internal-complibs/zlib-<ZLIB_VERSION>/ directory
+in the .github/workflows/ci-emscripten.yaml workflow file.
 
-diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c b/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
+diff --git a/gzlib.c b/gzlib.c
 index fae202e..80606a6 100644
---- a/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
-+++ b/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
+--- a/gzlib.c
++++ b/gzlib.c
 @@ -2,7 +2,7 @@
   * Copyright (C) 2004, 2010, 2011, 2012, 2013 Mark Adler
   * For conditions of distribution and use, see copyright notice in zlib.h
@@ -19,10 +21,10 @@ index fae202e..80606a6 100644
 
  #if defined(_WIN32) && !defined(__BORLANDC__)
 
-diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzread.c b/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
+diff --git a/gzread.c b/gzread.c
 index bf4538e..afe6acd 100644
---- a/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
-+++ b/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
+--- a/gzread.c
++++ b/gzread.c
 @@ -2,7 +2,7 @@
   * Copyright (C) 2004, 2005, 2010, 2011, 2012, 2013 Mark Adler
   * For conditions of distribution and use, see copyright notice in zlib.h
@@ -34,10 +36,10 @@ index bf4538e..afe6acd 100644
  /* Local functions */
 
 
-diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c b/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
+diff --git a/gzwrite.c b/gzwrite.c
 index aa767fb..a87676f 100644
---- a/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
-+++ b/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
+--- a/gzwrite.c
++++ b/gzwrite.c
 @@ -2,7 +2,7 @@
   * Copyright (C) 2004, 2005, 2010, 2011, 2012, 2013 Mark Adler
   * For conditions of distribution and use, see copyright notice in zlib.h

From 1aa838bf4890a50691928352dbcf109cc7f3642e Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 21:32:03 +0530
Subject: [PATCH 21/43] Install test + runtime extras, disable pytest cache

---
 .github/workflows/ci-emscripten.yaml | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index 17b1f1c6..da337e23 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -81,11 +81,13 @@ jobs:
           pyodide venv .venv-pyodide
           source .venv-pyodide/bin/activate
 
-          # Install the built numcodecs WASM package
-          python -m pip install dist/*.whl
-          python -m pip install pytest pytest-cov coverage
+          # Install the built numcodecs WASM wheel and relevant dependencies
+          python -m pip install dist/*.whl[msgpack,crc32c,test,test_extras]
 
           # Change into a different directory before running tests to avoid
           # the test runner picking up the local numcodecs package
           cd docs
-          python -m pytest --pyargs numcodecs
+
+          # Don't use the cache provider plugin, as it doesn't currently work
+          # with Pyodide: https://github.com/pypa/cibuildwheel/issues/1966
+          python -m pytest -p no:cacheprovider --pyargs numcodecs

From 0638a919ec4b43e6de39408acc5d0fc2e709bbcb Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 7 Nov 2024 21:40:05 +0530
Subject: [PATCH 22/43] Fix shell expansion in installation command

---
 .github/workflows/ci-emscripten.yaml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index da337e23..102443a1 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -82,7 +82,8 @@ jobs:
           source .venv-pyodide/bin/activate
 
           # Install the built numcodecs WASM wheel and relevant dependencies
-          python -m pip install dist/*.whl[msgpack,crc32c,test,test_extras]
+          pip install $(ls dist/*.whl)"[msgpack,crc32c,test,test_extras]"
+          # TODO: get zfpy built in Pyodide and install it here
 
           # Change into a different directory before running tests to avoid
           # the test runner picking up the local numcodecs package

From 53f2096a5e8ae3b3f9fcc8779b1d6525291e66ca Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 19:47:47 +0530
Subject: [PATCH 23/43] Temporary: skip tests that require crc32c

---
 numcodecs/tests/test_checksum32.py | 40 ++++++++++++++++++++++++------
 1 file changed, 32 insertions(+), 8 deletions(-)

diff --git a/numcodecs/tests/test_checksum32.py b/numcodecs/tests/test_checksum32.py
index 7aad9da3..cb313a08 100644
--- a/numcodecs/tests/test_checksum32.py
+++ b/numcodecs/tests/test_checksum32.py
@@ -3,7 +3,19 @@
 import numpy as np
 import pytest
 
-from numcodecs.checksum32 import CRC32, CRC32C, Adler32
+from numcodecs.checksum32 import CRC32, Adler32
+
+# Try to import CRC32C, skip entire module if not available
+# This is a temporary skip until we update the crc32c package
+# in Pyodide and make a release.
+try:
+    import crc32c  # noqa: F401, I001
+    from numcodecs.checksum32 import CRC32C
+
+    HAS_CRC32C = True
+except ImportError:  # pragma: no cover
+    HAS_CRC32C = False
+
 from numcodecs.tests.common import (
     check_backwards_compatibility,
     check_config,
@@ -32,15 +44,23 @@
     np.random.randint(-(2**63), -(2**63) + 20, size=1000, dtype='i8').view('m8[m]'),
 ]
 
+# Initialize base codecs
 codecs = [
     CRC32(),
     CRC32(location="end"),
-    CRC32C(location="start"),
-    CRC32C(),
     Adler32(),
     Adler32(location="end"),
 ]
 
+# Add CRC32C codecs if available
+if HAS_CRC32C:
+    codecs.extend(
+        [
+            CRC32C(location="start"),
+            CRC32C(),
+        ]
+    )
+
 
 @pytest.mark.parametrize(("codec", "arr"), itertools.product(codecs, arrays))
 def test_encode_decode(codec, arr):
@@ -84,25 +104,28 @@ def test_err_encode_list(codec):
 def test_err_location():
     with pytest.raises(ValueError):
         CRC32(location="foo")
-    with pytest.raises(ValueError):
-        CRC32C(location="foo")
     with pytest.raises(ValueError):
         Adler32(location="foo")
+    if HAS_CRC32C:
+        with pytest.raises(ValueError):
+            CRC32C(location="foo")
 
 
 def test_repr():
     check_repr("CRC32(location='start')")
-    check_repr("CRC32C(location='start')")
     check_repr("Adler32(location='start')")
     check_repr("CRC32(location='end')")
-    check_repr("CRC32C(location='end')")
     check_repr("Adler32(location='end')")
+    if HAS_CRC32C:
+        check_repr("CRC32C(location='start')")
+        check_repr("CRC32C(location='end')")
 
 
 def test_backwards_compatibility():
     check_backwards_compatibility(CRC32.codec_id, arrays, [CRC32()])
     check_backwards_compatibility(Adler32.codec_id, arrays, [Adler32()])
-    check_backwards_compatibility(CRC32C.codec_id, arrays, [CRC32C()])
+    if HAS_CRC32C:
+        check_backwards_compatibility(CRC32C.codec_id, arrays, [CRC32C()])
 
 
 @pytest.mark.parametrize("codec", codecs)
@@ -123,6 +146,7 @@ def test_err_out_too_small(codec):
         codec.decode(codec.encode(arr), out)
 
 
+@pytest.mark.skipif(not HAS_CRC32C, reason="CRC32C not available")
 def test_crc32c_checksum():
     arr = np.arange(0, 64, dtype="uint8")
     buf = CRC32C(location="end").encode(arr)

From bc9b6dc099a710b1b1eda0cb39abb98bc39230d1 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 19:57:09 +0530
Subject: [PATCH 24/43] Temporary: skip `crc32c` dependency until available

---
 pyproject.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pyproject.toml b/pyproject.toml
index 8e1a03f8..1d0b9711 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -71,7 +71,7 @@ pcodec = [
     "pcodec>=0.2,<0.3",
 ]
 crc32c = [
-    "crc32c>=2.7",
+    "crc32c>=2.7; sys_platform == 'emscripten'",
 ]
 
 [project.entry-points."zarr.codecs"]

From 5a4e23404d5a9e4116cee673e889def6b1f5da6d Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 20:02:14 +0530
Subject: [PATCH 25/43] Update ci-emscripten.yaml

---
 .github/workflows/ci-emscripten.yaml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index 102443a1..be3b813e 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -85,6 +85,9 @@ jobs:
           pip install $(ls dist/*.whl)"[msgpack,crc32c,test,test_extras]"
           # TODO: get zfpy built in Pyodide and install it here
 
+          # For tests in test_zarr3.py
+          pip install zarr
+
           # Change into a different directory before running tests to avoid
           # the test runner picking up the local numcodecs package
           cd docs

From 4ce9e4d081f374555a06182a47874c1796167dbc Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 20:06:26 +0530
Subject: [PATCH 26/43] Fix zarr installation order to collect more tests

---
 .github/workflows/ci-emscripten.yaml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index be3b813e..634616fb 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -81,13 +81,13 @@ jobs:
           pyodide venv .venv-pyodide
           source .venv-pyodide/bin/activate
 
+          # For tests in test_zarr3.py
+          pip install zarr
+
           # Install the built numcodecs WASM wheel and relevant dependencies
           pip install $(ls dist/*.whl)"[msgpack,crc32c,test,test_extras]"
           # TODO: get zfpy built in Pyodide and install it here
 
-          # For tests in test_zarr3.py
-          pip install zarr
-
           # Change into a different directory before running tests to avoid
           # the test runner picking up the local numcodecs package
           cd docs

From d6c43efe5176c300917a6dd4e5e0d94d63c0e1c4 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 21:12:48 +0530
Subject: [PATCH 27/43] Force install zarr version 3 for CI

---
 .github/workflows/ci-emscripten.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index 634616fb..13ee05ce 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -82,7 +82,7 @@ jobs:
           source .venv-pyodide/bin/activate
 
           # For tests in test_zarr3.py
-          pip install zarr
+          pip install zarr==3.0.0b1
 
           # Install the built numcodecs WASM wheel and relevant dependencies
           pip install $(ls dist/*.whl)"[msgpack,crc32c,test,test_extras]"

From 55a2690033c176609d62be7040f1dfb765f21734 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 21:13:04 +0530
Subject: [PATCH 28/43] Skip a test that requires threads

---
 numcodecs/tests/test_zarr3.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/numcodecs/tests/test_zarr3.py b/numcodecs/tests/test_zarr3.py
index 78d824e2..d0bd7613 100644
--- a/numcodecs/tests/test_zarr3.py
+++ b/numcodecs/tests/test_zarr3.py
@@ -4,6 +4,7 @@
 import pytest
 
 import numcodecs.zarr3
+from numcodecs.tests.common import is_wasm
 
 zarr = pytest.importorskip("zarr")
 
@@ -46,6 +47,7 @@ def test_docstring(codec_class: type[numcodecs.zarr3._NumcodecsCodec]):
     assert "See :class:`numcodecs." in codec_class.__doc__
 
 
+@pytest.mark.skipif(is_wasm, reason="Threads are not supported in Pyodide/WASM")
 @pytest.mark.parametrize(
     "codec_class",
     [

From 9a0482b47b0909f7bd6e9b815ad0223f8f321205 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 21:14:02 +0530
Subject: [PATCH 29/43] Temporary: skip Zarr's async functionality tests

---
 numcodecs/tests/test_zarr3.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/numcodecs/tests/test_zarr3.py b/numcodecs/tests/test_zarr3.py
index d0bd7613..4003f066 100644
--- a/numcodecs/tests/test_zarr3.py
+++ b/numcodecs/tests/test_zarr3.py
@@ -78,6 +78,8 @@ def test_generic_codec_class(store: Store, codec_class: type[numcodecs.zarr3._Nu
     np.testing.assert_array_equal(data, a[:, :])
 
 
+# TODO: undo skips here when we can test async code
+@pytest.mark.skipif(is_wasm, reason="testing async code not yet supported in Pyodide/WASM")
 @pytest.mark.parametrize(
     ("codec_class", "codec_config"),
     [
@@ -116,6 +118,8 @@ def test_generic_filter(
     np.testing.assert_array_equal(data, a[:, :])
 
 
+# TODO: undo skips here when we can test async code
+@pytest.mark.skipif(is_wasm, reason="testing async code not yet supported in Pyodide/WASM")
 def test_generic_filter_bitround(store: Store):
     data = np.linspace(0, 1, 256, dtype="float32").reshape((16, 16))
 
@@ -134,6 +138,8 @@ def test_generic_filter_bitround(store: Store):
     assert np.allclose(data, a[:, :], atol=0.1)
 
 
+# TODO: undo skips here when we can test async code
+@pytest.mark.skipif(is_wasm, reason="testing async code not yet supported in Pyodide/WASM")
 def test_generic_filter_quantize(store: Store):
     data = np.linspace(0, 10, 256, dtype="float32").reshape((16, 16))
 
@@ -152,6 +158,8 @@ def test_generic_filter_quantize(store: Store):
     assert np.allclose(data, a[:, :], atol=0.001)
 
 
+# TODO: undo skips here when we can test async code
+@pytest.mark.skipif(is_wasm, reason="testing async code not yet supported in Pyodide/WASM")
 def test_generic_filter_packbits(store: Store):
     data = np.zeros((16, 16), dtype="bool")
     data[0:4, :] = True
@@ -181,6 +189,8 @@ def test_generic_filter_packbits(store: Store):
         )
 
 
+# TODO: undo skips here when we can test async code
+@pytest.mark.skipif(is_wasm, reason="testing async code not yet supported in Pyodide/WASM")
 @pytest.mark.parametrize(
     "codec_class",
     [

From bf4249191e538c9d8462eef168ebed643e2c241d Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 21:19:27 +0530
Subject: [PATCH 30/43] Guard import of `multiprocessing`

---
 numcodecs/tests/test_entrypoints_backport.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/numcodecs/tests/test_entrypoints_backport.py b/numcodecs/tests/test_entrypoints_backport.py
index 21f245aa..42d895da 100644
--- a/numcodecs/tests/test_entrypoints_backport.py
+++ b/numcodecs/tests/test_entrypoints_backport.py
@@ -1,7 +1,11 @@
 import importlib
 import os.path
 import sys
-from multiprocessing import Process
+
+from numcodecs.tests.common import is_wasm
+
+if not is_wasm:  # pragma: no cover
+    from multiprocessing import Process
 
 import pytest
 

From 9924e8e85b325ba32d40f05f98d00478ac187891 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 21:33:17 +0530
Subject: [PATCH 31/43] Revert "Guard import of `multiprocessing`"

This reverts commit bf4249191e538c9d8462eef168ebed643e2c241d.
---
 numcodecs/tests/test_entrypoints_backport.py | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/numcodecs/tests/test_entrypoints_backport.py b/numcodecs/tests/test_entrypoints_backport.py
index 42d895da..21f245aa 100644
--- a/numcodecs/tests/test_entrypoints_backport.py
+++ b/numcodecs/tests/test_entrypoints_backport.py
@@ -1,11 +1,7 @@
 import importlib
 import os.path
 import sys
-
-from numcodecs.tests.common import is_wasm
-
-if not is_wasm:  # pragma: no cover
-    from multiprocessing import Process
+from multiprocessing import Process
 
 import pytest
 

From 08472414d5f57fe5e1e3c3a2bf98ab23c0ecaf6e Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 21:34:51 +0530
Subject: [PATCH 32/43] Bump verbosity when running test suite

---
 .github/workflows/ci-emscripten.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index 13ee05ce..75a5d0c8 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -94,4 +94,4 @@ jobs:
 
           # Don't use the cache provider plugin, as it doesn't currently work
           # with Pyodide: https://github.com/pypa/cibuildwheel/issues/1966
-          python -m pytest -p no:cacheprovider --pyargs numcodecs
+          python -m pytest -p no:cacheprovider -svra --pyargs numcodecs

From b6713ca6b4942ac226dac051b943362c74d0ef68 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 21:35:35 +0530
Subject: [PATCH 33/43] Skip codec entrypoint test that requires processes

---
 numcodecs/tests/test_entrypoints_backport.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/numcodecs/tests/test_entrypoints_backport.py b/numcodecs/tests/test_entrypoints_backport.py
index 21f245aa..57c21caa 100644
--- a/numcodecs/tests/test_entrypoints_backport.py
+++ b/numcodecs/tests/test_entrypoints_backport.py
@@ -6,6 +6,7 @@
 import pytest
 
 import numcodecs.registry
+from numcodecs.tests.common import is_wasm
 
 if not importlib.util.find_spec("importlib_metadata").loader:  # pragma: no cover
     pytest.skip(
@@ -28,6 +29,7 @@ def get_entrypoints_with_importlib_metadata_loaded():
     assert cls.codec_id == "test"
 
 
+@pytest.mark.skipif(is_wasm, reason="Threads are not supported in Pyodide/WASM")
 def test_entrypoint_codec_with_importlib_metadata():
     p = Process(target=get_entrypoints_with_importlib_metadata_loaded)
     p.start()

From a5907859ac2350cb65ea82ff9547798cde4ff8a7 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 21:54:49 +0530
Subject: [PATCH 34/43] Let `pytest` inherit config for doctests

---
 .github/workflows/ci-emscripten.yaml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index 75a5d0c8..0e3b459c 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -89,8 +89,11 @@ jobs:
           # TODO: get zfpy built in Pyodide and install it here
 
           # Change into a different directory before running tests to avoid
-          # the test runner picking up the local numcodecs package
+          # the test runner picking up the local numcodecs package. Plus let
+          # pytest inherit its configuration from pyproject.toml.
           cd docs
+          cp ../pyproject.toml .
+
 
           # Don't use the cache provider plugin, as it doesn't currently work
           # with Pyodide: https://github.com/pypa/cibuildwheel/issues/1966

From 3e1a9689d2abd6929e3df219fdca08d4b24eb0e1 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 22:23:53 +0530
Subject: [PATCH 35/43] Revert "Let `pytest` inherit config for doctests"

This reverts commit a5907859ac2350cb65ea82ff9547798cde4ff8a7.
---
 .github/workflows/ci-emscripten.yaml | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index 0e3b459c..75a5d0c8 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -89,11 +89,8 @@ jobs:
           # TODO: get zfpy built in Pyodide and install it here
 
           # Change into a different directory before running tests to avoid
-          # the test runner picking up the local numcodecs package. Plus let
-          # pytest inherit its configuration from pyproject.toml.
+          # the test runner picking up the local numcodecs package
           cd docs
-          cp ../pyproject.toml .
-
 
           # Don't use the cache provider plugin, as it doesn't currently work
           # with Pyodide: https://github.com/pypa/cibuildwheel/issues/1966

From e4f66703c65b1c877944324de2c9fd14ee9c1604 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Fri, 8 Nov 2024 23:14:59 +0530
Subject: [PATCH 36/43] Ignore a doctest that uses threads within the file

---
 numcodecs/zarr3.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/numcodecs/zarr3.py b/numcodecs/zarr3.py
index 811ab501..c5ff60a4 100644
--- a/numcodecs/zarr3.py
+++ b/numcodecs/zarr3.py
@@ -24,6 +24,19 @@
 
 from __future__ import annotations
 
+# Short workaround for skipping the doctest above in a WASM environment
+# compiled via Emscripten where threads are not available, and accessing
+# the pytest config has no effect and ignoring warnings does not work.
+try:
+    import pytest
+
+    from numcodecs.tests.common import is_wasm
+
+    if is_wasm:
+        pytest.skip("zarr3 doctests not supported in WASM", allow_module_level=True)
+except (ImportError, ModuleNotFoundError):  # not running tests
+    pass
+
 import asyncio
 import math
 from collections.abc import Callable

From dacbd09644ba71bc0c7340b2026d505d1239c064 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 1 Jan 2025 23:08:52 +0530
Subject: [PATCH 37/43] Fix skip reason error message for entrypoint test

---
 numcodecs/tests/test_entrypoints_backport.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/numcodecs/tests/test_entrypoints_backport.py b/numcodecs/tests/test_entrypoints_backport.py
index ee2493e5..ded5d33a 100644
--- a/numcodecs/tests/test_entrypoints_backport.py
+++ b/numcodecs/tests/test_entrypoints_backport.py
@@ -30,7 +30,7 @@ def get_entrypoints_with_importlib_metadata_loaded():
     assert cls.codec_id == "test"
 
 
-@pytest.mark.skipif(is_wasm, reason="Threads are not supported in Pyodide/WASM")
+@pytest.mark.skipif(is_wasm, reason="Spawning processes is not supported in Pyodide/WASM")
 def test_entrypoint_codec_with_importlib_metadata():
     p = Process(target=get_entrypoints_with_importlib_metadata_loaded)
     p.start()

From 6caf0763dbaa7bfced83bc1af3c10586b2af18bc Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 1 Jan 2025 23:09:24 +0530
Subject: [PATCH 38/43] `crc32c` >=2.7 is now available with Pyodide 0.27.0

---
 pyproject.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pyproject.toml b/pyproject.toml
index 1137e129..7dfadb5d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -70,7 +70,7 @@ pcodec = [
     "pcodec>=0.3,<0.4",
 ]
 crc32c = [
-    "crc32c>=2.7; sys_platform == 'emscripten'",
+    "crc32c>=2.7",
 ]
 
 [project.entry-points."zarr.codecs"]

From 85252f4ba213cd0f1c378e236f491242a01dd1d7 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 1 Jan 2025 23:09:45 +0530
Subject: [PATCH 39/43] Bump to Pyodide version 0.27.0

---
 .github/workflows/ci-emscripten.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index 75a5d0c8..c7f7b302 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -15,12 +15,12 @@ env:
   DISABLE_NUMCODECS_AVX2: 1
   DISABLE_NUMCODECS_SSE2: 1
   # Common environment variables for both build and test jobs
-  PYODIDE_VERSION: 0.27.0a2
+  PYODIDE_VERSION: 0.27.0
   # PYTHON_VERSION and EMSCRIPTEN_VERSION are determined by PYODIDE_VERSION.
   # The appropriate versions can be found in the Pyodide repodata.json
   # "info" field, or in Makefile.envs:
   # https://github.com/pyodide/pyodide/blob/main/Makefile.envs#L2
-  PYTHON_VERSION: 3.12.1
+  PYTHON_VERSION: 3.12 # any 3.12.x version works
   EMSCRIPTEN_VERSION: 3.1.58
   NODE_VERSION: 20
 

From 6b8d139f065fd5f865c819848d67e588aaddd67d Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 1 Jan 2025 23:15:28 +0530
Subject: [PATCH 40/43] Clean up changes

---
 numcodecs/tests/test_checksum32.py                     | 1 -
 numcodecs/tests/test_zarr3.py                          | 3 ++-
 tools/ci/patches/0002-add-missing-unistd-headers.patch | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/numcodecs/tests/test_checksum32.py b/numcodecs/tests/test_checksum32.py
index 4eece788..9bdc25cb 100644
--- a/numcodecs/tests/test_checksum32.py
+++ b/numcodecs/tests/test_checksum32.py
@@ -39,7 +39,6 @@
     np.random.randint(-(2**63), -(2**63) + 20, size=1000, dtype='i8').view('m8[m]'),
 ]
 
-# Initialize base codecs
 codecs = [
     CRC32(),
     CRC32(location="end"),
diff --git a/numcodecs/tests/test_zarr3.py b/numcodecs/tests/test_zarr3.py
index f2523d1c..68febd82 100644
--- a/numcodecs/tests/test_zarr3.py
+++ b/numcodecs/tests/test_zarr3.py
@@ -5,7 +5,6 @@
 import numpy as np
 import pytest
 
-import numcodecs.zarr3
 from numcodecs.tests.common import is_wasm
 
 if TYPE_CHECKING:  # pragma: no cover
@@ -16,6 +15,8 @@
 import zarr.storage
 from zarr.core.common import JSON
 
+import numcodecs.zarr3
+
 pytestmark = [
     pytest.mark.skipif(zarr.__version__ < "3.0.0", reason="zarr 3.0.0 or later is required"),
     pytest.mark.filterwarnings("ignore:Codec 'numcodecs.*' not configured in config.*:UserWarning"),
diff --git a/tools/ci/patches/0002-add-missing-unistd-headers.patch b/tools/ci/patches/0002-add-missing-unistd-headers.patch
index e7eb430e..7bbc88c1 100644
--- a/tools/ci/patches/0002-add-missing-unistd-headers.patch
+++ b/tools/ci/patches/0002-add-missing-unistd-headers.patch
@@ -48,4 +48,4 @@ index aa767fb..a87676f 100644
 +#include <unistd.h>
  #include "gzguts.h"
 
- /* Local functions */
\ No newline at end of file
+ /* Local functions */

From c3e346a01177410a008652948e7c6937760ec87c Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Tue, 18 Feb 2025 15:31:34 +0530
Subject: [PATCH 41/43] Bring back code-style quoting

Co-authored-by: Josh Moore <josh@openmicroscopy.org>
---
 docs/contributing.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/contributing.rst b/docs/contributing.rst
index d6bb5b0d..d323926f 100644
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@@ -111,7 +111,7 @@ Creating a branch
 Before you do any new work or submit a pull request, please open an issue on GitHub to
 report the bug or propose the feature you'd like to add.
 
-It's best to create a new, separate branch for each piece of work you want to do. E.g.:
+It's best to create a new, separate branch for each piece of work you want to do. E.g.::
 
     git fetch upstream
     git checkout -b shiny-new-feature upstream/main

From 2e0e75933c8d51a930093b8cf60aed7bce5356d7 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Wed, 26 Feb 2025 19:24:10 +0530
Subject: [PATCH 42/43] Update to Pyodide 0.27.3, add `pcodec`

---
 .github/workflows/ci-emscripten.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/ci-emscripten.yaml b/.github/workflows/ci-emscripten.yaml
index c7f7b302..af53ad24 100644
--- a/.github/workflows/ci-emscripten.yaml
+++ b/.github/workflows/ci-emscripten.yaml
@@ -15,7 +15,7 @@ env:
   DISABLE_NUMCODECS_AVX2: 1
   DISABLE_NUMCODECS_SSE2: 1
   # Common environment variables for both build and test jobs
-  PYODIDE_VERSION: 0.27.0
+  PYODIDE_VERSION: 0.27.3
   # PYTHON_VERSION and EMSCRIPTEN_VERSION are determined by PYODIDE_VERSION.
   # The appropriate versions can be found in the Pyodide repodata.json
   # "info" field, or in Makefile.envs:
@@ -85,7 +85,7 @@ jobs:
           pip install zarr==3.0.0b1
 
           # Install the built numcodecs WASM wheel and relevant dependencies
-          pip install $(ls dist/*.whl)"[msgpack,crc32c,test,test_extras]"
+          pip install $(ls dist/*.whl)"[msgpack,crc32c,pcodec,test,test_extras]"
           # TODO: get zfpy built in Pyodide and install it here
 
           # Change into a different directory before running tests to avoid

From 4e6f2b6d5915cae6d9efc127ec9f2e0cb1553bf8 Mon Sep 17 00:00:00 2001
From: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Date: Thu, 27 Feb 2025 15:48:10 +0530
Subject: [PATCH 43/43] Fix drop in coverage

---
 numcodecs/zarr3.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/numcodecs/zarr3.py b/numcodecs/zarr3.py
index 2b913091..58198dcd 100644
--- a/numcodecs/zarr3.py
+++ b/numcodecs/zarr3.py
@@ -33,7 +33,7 @@
 
     from numcodecs.tests.common import is_wasm
 
-    if is_wasm:
+    if is_wasm:  # pragma: no cover
         pytest.skip("zarr3 doctests not supported in WASM", allow_module_level=True)
 except (ImportError, ModuleNotFoundError):  # not running tests
     pass