[Bug]: Content of version 2.8.2 on PyPI changed unexpectedly

[benjamin-heasly]

What happened?

I have local code and automated builds for a project that uses pynwb as a dependency. I use a mix of Conda and pip to resolve eg pynwb==2.8.2.

Builds and local scripts that depend on pynwb==2.8.2 were working up until 17 January 2025. Starting on 3 February 2025 the same automated builds started failing. There was no change to the build process, but apparently a change to the pynwb==2.8.2 obtained from PyPi.

This required an unexpected upgrade to pynwb==2.8.3 to work around.

Steps to Reproduce

Difficult to reproduce, as this behavior has changed unexpectedly over time.

The command to install would be

pip install pynwb==2.8.2`


Then, a script that attempts pynwb imports will fail, for example:


from pynwb import NWBHDF5IO, NWBFile

Traceback

Traceback (most recent call last):
  File "/home/runner/work/steps-and-pipelines/steps-and-pipelines/steps/synthesis/test/test_synthesis.py", line 7, in <module>
    from pynwb import NWBHDF5IO, NWBFile
  File "/usr/share/miniconda/envs/steps-admin/lib/python3.11/site-packages/pynwb/__init__.py", line 413, in <module>
    from . import io as __io  # noqa: F401,E402
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/steps-admin/lib/python3.11/site-packages/pynwb/io/__init__.py", line 1, in <module>
    from . import base as __base
  File "/usr/share/miniconda/envs/steps-admin/lib/python3.11/site-packages/pynwb/io/base.py", line 3, in <module>
    from .core import NWBContainerMapper
  File "/usr/share/miniconda/envs/steps-admin/lib/python3.11/site-packages/pynwb/io/core.py", line 1, in <module>
    from hdmf.build import ObjectMapper, RegionBuilder
ImportError: cannot import name 'RegionBuilder' from 'hdmf.build' (/usr/share/miniconda/envs/steps-admin/lib/python3.11/site-packages/hdmf/build/__init__.py)

Operating System

Linux

Python Executable

Conda

Python Version

3.11

Package Versions

Please don't change the content of versioned distributions, after they are made public on PyPI. This can have ripple effects and defeats the purpose of versioning!

Better to create a new version (version numbers are unlimited and cheap!) and to report the reason as a release note and/or Issue.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this bug was not already reported?

[t-b]

I use a mix of Conda and pip to resolve eg pynwb==2.8.2.

That's already a danger zone.

Builds and local scripts that depend on pynwb==2.8.2 were working up until 17 January > 2025. Starting on 3 February 2025 the same automated builds started failing. There was > no change to the build process, but apparently a change to the pynwb==2.8.2 obtained
from PyPi.

Do you pin all dependencies of pynwb or only pnwb? I had an issue a while ago where a newer hdmf version was released and when that broke the old pynwb version. And looks like hdmf got also an update on Jan 22 2025 1.

[stephprince]

Hi @benjamin-heasly, thanks for raising this issue.

As @t-b mentioned above, I believe the ImportError you reported began to occur due to a new release of hdmf (one of the main pynwb dependencies) on Jan 22, 2025. The pynwb 2.8.2 version contents on PyPi should not have been changed.

This required an unexpected upgrade to pynwb==2.8.3 to work around.

It sounds like you were able to resolve this issue by updating to the latest version of pynwb. I wanted to note that the other option is to pin the hdmf version to <4 if you would like to continue to use pynwb 2.8.2 (or earlier pynwb versions):

pip install "hdmf<4" "pynwb==2.8.2”

[benjamin-heasly]

Thank you, this makes sense to me now and I'm glad that the content of 2.8.2 didn't change! Thanks for pointing out the hdmf dependency.

I'll restate my understanding, in case this is useful to someone else who finds this issue in a search.

pynwb depends on hdmf. In particular pynwb==2.8.2 declares a dependency on hdmf>=3.14.3 which has no upper bound.

pynwb==2.8.2 is not compatible with hdmf 4, so when that version came out in January 2025 it became possible to create broken installations with pynwb==2.8.2 and hdmf 4. The dynamic element was pip choosing the new, latest version of hdmf, rather than a change in the pynwb package contents.

Users can defensively pin hdmf to an earlier version, for example hdmf==3.14.3, in order to create working installations. "working" is no longer the default behavior of the package installer.

[t-b]

@stephprince Not sure I agree with just closing the issue. Wouldn't it make sense to change the dependency on hdmf from "hdmf>=3.14.5", to "hdmf>=3.14.5,<4", to prevent further issues like this for newer releases?

[stephprince]

@t-b since hdmf 4.0 is supported by pynwb>=2.8.3, the hdmf dependency should not be restricted to "hdmf>=3.14.5,<4" for newer pynwb releases.

However to prevent similar issues with future major releases of hdmf, there are a couple potential options:

1. We could start setting an upper bound for hdmf a few releases in the future. We want to avoid in general setting upper bounds for libraries, but since hdmf and pynwb are closely related and follow a similar release cadence this may be appropriate here.
2. We could tightly couple hdmf and pynwb releases so that each version of pynwb requires a specific version of hdmf.

For now I made an issue to add additional documentation about hdmf/pynwb compatibility (#2034), but we can discuss these options further.

[h-mayorquin]

I vote for some version of 1.
Having a ceiling for major version of hdmf sounds like a good idea overall. That is:

• Ceiling by default on the bound of the new hdmf major release
• Ceiling on a minor release only when an error is discovered.

What are the drawbacks of such a proposal?

[stephprince]

This article, "Should You Use Upper Bound Version Constraints", discusses some of the general drawbacks of upper bounds. However it also mentions that two related libraries that are maintained and released together can be an ok exception.

I think in this case we could set the upper bound 2-3 major releases in advance, and as long as we deprecate functions and classes in HDMF before removing them entirely, this should be ok for future library installation and use.