diff --git a/.github/workflows/check_external_links.yml b/.github/workflows/check_sphinx_links.yml similarity index 82% rename from .github/workflows/check_external_links.yml rename to .github/workflows/check_sphinx_links.yml index 1c709ba79..e1eddb97a 100644 --- a/.github/workflows/check_external_links.yml +++ b/.github/workflows/check_sphinx_links.yml @@ -1,4 +1,4 @@ -name: Check Sphinx external links +name: Check Sphinx links on: pull_request: schedule: @@ -6,7 +6,7 @@ on: workflow_dispatch: jobs: - check-external-links: + check-sphinx-links: runs-on: ubuntu-latest steps: - name: Cancel non-latest runs @@ -31,5 +31,5 @@ jobs: python -m pip install -r requirements-doc.txt python -m pip install . - - name: Check Sphinx external links - run: sphinx-build -b linkcheck ./docs/source ./test_build + - name: Check Sphinx internal and external links + run: sphinx-build -W -b linkcheck ./docs/source ./test_build diff --git a/CHANGELOG.md b/CHANGELOG.md index f084ff0bd..4bd5e4fdd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ - For `NWBHDF5IO()`, change the default of arg `load_namespaces` from `False` to `True`. @bendichter [#1748](https://github.com/NeurodataWithoutBorders/pynwb/pull/1748) - Add `NWBHDF5IO.can_read()`. @bendichter [#1703](https://github.com/NeurodataWithoutBorders/pynwb/pull/1703) - Add `pynwb.get_nwbfile_version()`. @bendichter [#1703](https://github.com/NeurodataWithoutBorders/pynwb/pull/1703) +- Fix usage of the `validate` function in the `pynwb.testing.testh5io` classes and cache the spec by default in those classes. @rly [#1782](https://github.com/NeurodataWithoutBorders/pynwb/pull/1782) - Updated timeseries data checks to warn instead of error when reading invalid files. @stephprince [#1793](https://github.com/NeurodataWithoutBorders/pynwb/pull/1793) and [#1809](https://github.com/NeurodataWithoutBorders/pynwb/pull/1809) - Expose the offset, conversion and channel conversion parameters in `mock_ElectricalSeries`. @h-mayorquin [#1796](https://github.com/NeurodataWithoutBorders/pynwb/pull/1796) - Expose `starting_time` in `mock_ElectricalSeries`. @h-mayorquin [#1805](https://github.com/NeurodataWithoutBorders/pynwb/pull/1805) @@ -21,17 +22,19 @@ ### Bug fixes - Fix bug where namespaces were loaded in "w-" mode. @h-mayorquin [#1795](https://github.com/NeurodataWithoutBorders/pynwb/pull/1795) - Fix bug where pynwb version was reported as "unknown" to readthedocs @stephprince [#1810](https://github.com/NeurodataWithoutBorders/pynwb/pull/1810) +- Fixed bug to allow linking of `TimeSeries.data` by setting the `data` constructor argument to another `TimeSeries`. @oruebel [#1766](https://github.com/NeurodataWithoutBorders/pynwb/pull/1766) ### Documentation and tutorial enhancements - Add RemFile to streaming tutorial. @bendichter [#1761](https://github.com/NeurodataWithoutBorders/pynwb/pull/1761) - Fix typos and improve clarify throughout tutorials. @zm711 [#1825](https://github.com/NeurodataWithoutBorders/pynwb/pull/1825) +- Fix internal links in docstrings and tutorials. @stephprince [#1827](https://github.com/NeurodataWithoutBorders/pynwb/pull/1827) - Add Zarr IO tutorial @bendichter [#1834](https://github.com/NeurodataWithoutBorders/pynwb/pull/1834) ## PyNWB 2.5.0 (August 18, 2023) ### Enhancements and minor changes -- Add `TimeSeries.get_timestamps()`. @bendichter [#1741](https://github.com/NeurodataWithoutBorders/pynwb/pull/1741) -- Add `TimeSeries.get_data_in_units()`. @bendichter [#1745](https://github.com/NeurodataWithoutBorders/pynwb/pull/1745) +- Added `TimeSeries.get_timestamps()`. @bendichter [#1741](https://github.com/NeurodataWithoutBorders/pynwb/pull/1741) +- Added `TimeSeries.get_data_in_units()`. @bendichter [#1745](https://github.com/NeurodataWithoutBorders/pynwb/pull/1745) - Updated `ExternalResources` name change to `HERD`, along with HDMF 3.9.0 being the new minimum. @mavaylon1 [#1754](https://github.com/NeurodataWithoutBorders/pynwb/pull/1754) ### Documentation and tutorial enhancements @@ -44,15 +47,15 @@ ## PyNWB 2.4.0 (July 23, 2023) ### Enhancements and minor changes -- Add support for `ExternalResources`. @mavaylon1 [#1684](https://github.com/NeurodataWithoutBorders/pynwb/pull/1684) -- Update links for making a release. @mavaylon1 [#1720](https://github.com/NeurodataWithoutBorders/pynwb/pull/1720) +- Added support for `ExternalResources`. @mavaylon1 [#1684](https://github.com/NeurodataWithoutBorders/pynwb/pull/1684) +- Updated links for making a release. @mavaylon1 [#1720](https://github.com/NeurodataWithoutBorders/pynwb/pull/1720) ### Bug fixes - Fixed sphinx-gallery setting to correctly display index in the docs with sphinx-gallery>=0.11. @oruebel [#1733](https://github.com/NeurodataWithoutBorders/pynwb/pull/1733) ### Documentation and tutorial enhancements - Added thumbnail for Optogentics tutorial. @oruebel [#1729](https://github.com/NeurodataWithoutBorders/pynwb/pull/1729) -- Update and fix errors in tutorials. @bendichter @oruebel +- Updated and fixed errors in tutorials. @bendichter @oruebel ## PyNWB 2.3.3 (June 26, 2023) diff --git a/docs/Makefile b/docs/Makefile index 80492bbf2..26960f3e7 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -149,7 +149,7 @@ changes: @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck + $(SPHINXBUILD) -W -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." diff --git a/docs/gallery/advanced_io/h5dataio.py b/docs/gallery/advanced_io/h5dataio.py index fd3548e4b..3b4391655 100644 --- a/docs/gallery/advanced_io/h5dataio.py +++ b/docs/gallery/advanced_io/h5dataio.py @@ -34,7 +34,7 @@ #################### -# Normally if we create a :py:class:`~pynwb.file.TimeSeries` we would do +# Normally if we create a :py:class:`~pynwb.base.TimeSeries` we would do import numpy as np diff --git a/docs/gallery/advanced_io/plot_editing.py b/docs/gallery/advanced_io/plot_editing.py index e45e3b887..a371dc588 100644 --- a/docs/gallery/advanced_io/plot_editing.py +++ b/docs/gallery/advanced_io/plot_editing.py @@ -129,7 +129,7 @@ # Editing groups # -------------- # Editing of groups is not yet supported in PyNWB. -# To edit the attributes of a group, open the file and edit it using :py:mod:`h5py`: +# To edit the attributes of a group, open the file and edit it using ``h5py``: import h5py diff --git a/docs/gallery/advanced_io/streaming.py b/docs/gallery/advanced_io/streaming.py index 760e2da71..4bdc992b8 100644 --- a/docs/gallery/advanced_io/streaming.py +++ b/docs/gallery/advanced_io/streaming.py @@ -23,6 +23,11 @@ Now you can get the url of a particular NWB file using the dandiset ID and the path of that file within the dandiset. +.. note:: + + To learn more about the dandi API see the + `DANDI Python API docs `_ + """ # sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_streaming.png' diff --git a/docs/gallery/domain/images.py b/docs/gallery/domain/images.py index fafc1849f..fb33f3760 100644 --- a/docs/gallery/domain/images.py +++ b/docs/gallery/domain/images.py @@ -190,7 +190,7 @@ # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # # :py:class:`~pynwb.image.RGBAImage` is for storing data of color image with transparency. -# :py:attr:`~pynwb.image.RGBAImage.data` must be 3D where the first and second dimensions +# ``RGBAImage.data`` must be 3D where the first and second dimensions # represent x and y. The third dimension has length 4 and represents the RGBA value. # @@ -208,7 +208,7 @@ # ^^^^^^^^^^^^^^^^^^^^^^^^^^ # # :py:class:`~pynwb.image.RGBImage` is for storing data of RGB color image. -# :py:attr:`~pynwb.image.RGBImage.data` must be 3D where the first and second dimensions +# ``RGBImage.data`` must be 3D where the first and second dimensions # represent x and y. The third dimension has length 3 and represents the RGB value. # @@ -224,8 +224,7 @@ # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # # :py:class:`~pynwb.image.GrayscaleImage` is for storing grayscale image data. -# :py:attr:`~pynwb.image.GrayscaleImage.data` must be 2D where the first and second dimensions -# represent x and y. +# ``GrayscaleImage.data`` must be 2D where the first and second dimensions represent x and y. # gs_logo = GrayscaleImage( @@ -300,7 +299,7 @@ #################### # Here `data` contains the (0-indexed) index of the displayed image as they are ordered -# in the :py:class:`~pynwb.base.ImageReference`. +# in the :py:class:`~pynwb.base.ImageReferences`. # # Writing the images to an NWB File # --------------------------------------- diff --git a/docs/gallery/domain/ophys.py b/docs/gallery/domain/ophys.py index b8ddb1ae5..277e408db 100644 --- a/docs/gallery/domain/ophys.py +++ b/docs/gallery/domain/ophys.py @@ -540,7 +540,7 @@ # Data arrays are read passively from the file. # Calling the data attribute on a :py:class:`~pynwb.base.TimeSeries` # such as a :py:class:`~pynwb.ophys.RoiResponseSeries` does not read the data -# values, but presents an :py:class:`~h5py` object that can be indexed to read data. +# values, but presents an ``h5py`` object that can be indexed to read data. # You can use the ``[:]`` operator to read the entire data array into memory. # Load and print all the data values of the :py:class:`~pynwb.ophys.RoiResponseSeries` # object representing the fluorescence data. @@ -558,7 +558,7 @@ # # It is often preferable to read only a portion of the data. To do this, index # or slice into the data attribute just like if you were indexing or slicing a -# :py:class:`~numpy` array. +# :py:mod:`numpy` array. # # The following code prints elements ``0:10`` in the first dimension (time) # and ``0:3`` (ROIs) in the second dimension from the fluorescence data we have written. diff --git a/docs/gallery/domain/plot_icephys.py b/docs/gallery/domain/plot_icephys.py index 2e7f51a23..8bb456b7f 100644 --- a/docs/gallery/domain/plot_icephys.py +++ b/docs/gallery/domain/plot_icephys.py @@ -351,7 +351,7 @@ ##################################################################### # .. note:: Since :py:meth:`~pynwb.file.NWBFile.add_intracellular_recording` can automatically add # the objects to the NWBFile we do not need to separately call -# :py:meth:`~pynwb.file.NWBFile.add_stimulus` and :py:meth:`~pynwb.file.NWBFile.add_acquistion` +# :py:meth:`~pynwb.file.NWBFile.add_stimulus` and :py:meth:`~pynwb.file.NWBFile.add_acquisition` # to add our stimulus and response, but it is still fine to do so. # # .. note:: The ``id`` parameter in the call is optional and if the ``id`` is omitted then PyNWB will @@ -549,8 +549,7 @@ # .. note:: The same process applies to all our other tables as well. We can use the # corresponding :py:meth:`~pynwb.file.NWBFile.get_intracellular_recordings`, # :py:meth:`~pynwb.file.NWBFile.get_icephys_sequential_recordings`, -# :py:meth:`~pynwb.file.NWBFile.get_icephys_repetitions`, and -# :py:meth:`~pynwb.file.NWBFile.get_icephys_conditions` functions instead. +# :py:meth:`~pynwb.file.NWBFile.get_icephys_repetitions` functions instead. # In general, we can always use the get functions instead of accessing the property # of the file. # @@ -561,7 +560,7 @@ # # Add a single simultaneous recording consisting of a set of intracellular recordings. # Again, setting the id for a simultaneous recording is optional. The recordings -# argument of the :py:meth:`~pynwb.file.NWBFile.add_simultaneous_recording` function +# argument of the :py:meth:`~pynwb.file.NWBFile.add_icephys_simultaneous_recording` function # here is simply a list of ints with the indices of the corresponding rows in # the :py:class:`~pynwb.icephys.IntracellularRecordingsTable` # @@ -618,7 +617,7 @@ # Add a single sequential recording consisting of a set of simultaneous recordings. # Again, setting the id for a sequential recording is optional. Also this table is # optional and will be created automatically by NWBFile. The ``simultaneous_recordings`` -# argument of the :py:meth:`~pynwb.file.NWBFile.add_sequential_recording` function +# argument of the :py:meth:`~pynwb.file.NWBFile.add_icephys_sequential_recording` function # here is simply a list of ints with the indices of the corresponding rows in # the :py:class:`~pynwb.icephys.SimultaneousRecordingsTable`. @@ -633,7 +632,7 @@ # Add a single repetition consisting of a set of sequential recordings. Again, setting # the id for a repetition is optional. Also this table is optional and will be created # automatically by NWBFile. The ``sequential_recordings argument`` of the -# :py:meth:`~pynwb.file.NWBFile.add_sequential_recording` function here is simply +# :py:meth:`~pynwb.file.NWBFile.add_icephys_repetition` function here is simply # a list of ints with the indices of the corresponding rows in # the :py:class:`~pynwb.icephys.SequentialRecordingsTable`. @@ -646,7 +645,7 @@ # Add a single experimental condition consisting of a set of repetitions. Again, # setting the id for a condition is optional. Also this table is optional and # will be created automatically by NWBFile. The ``repetitions`` argument of -# the :py:meth:`~pynwb.file.NWBFile.add_icephys_condition` function again is +# the :py:meth:`~pynwb.file.NWBFile.add_icephys_experimental_condition` function again is # simply a list of ints with the indices of the correspondingto rows in the # :py:class:`~pynwb.icephys.RepetitionsTable`. diff --git a/docs/gallery/general/add_remove_containers.py b/docs/gallery/general/add_remove_containers.py index 90ed8f324..86aa373b2 100644 --- a/docs/gallery/general/add_remove_containers.py +++ b/docs/gallery/general/add_remove_containers.py @@ -78,7 +78,7 @@ # have raw data and processed data in the same NWB file and you want to create a new NWB file with all the contents of # the original file except for the raw data for sharing with collaborators. # -# To remove existing containers, use the :py:class:`~hdmf.utils.LabelledDict.pop` method on any +# To remove existing containers, use the :py:meth:`~hdmf.utils.LabelledDict.pop` method on any # :py:class:`~hdmf.utils.LabelledDict` object, such as ``NWBFile.acquisition``, ``NWBFile.processing``, # ``NWBFile.analysis``, ``NWBFile.processing``, ``NWBFile.scratch``, ``NWBFile.devices``, ``NWBFile.stimulus``, # ``NWBFile.stimulus_template``, ``NWBFile.electrode_groups``, ``NWBFile.imaging_planes``, diff --git a/docs/gallery/general/object_id.py b/docs/gallery/general/object_id.py index 206142715..a4de45625 100644 --- a/docs/gallery/general/object_id.py +++ b/docs/gallery/general/object_id.py @@ -10,7 +10,7 @@ unique and used widely across computing platforms as if they are unique. The object ID of an NWB container object can be accessed using the -:py:meth:`~hdmf.container.AbstractContainer.object_id` method. +:py:attr:`~hdmf.container.AbstractContainer.object_id` method. .. _UUID: https://en.wikipedia.org/wiki/Universally_unique_identifier diff --git a/docs/gallery/general/plot_file.py b/docs/gallery/general/plot_file.py index 2175f8e9e..5c59abf8d 100644 --- a/docs/gallery/general/plot_file.py +++ b/docs/gallery/general/plot_file.py @@ -24,7 +24,7 @@ * :ref:`modules_overview`, i.e., objects for storing and grouping analyses, and * experiment metadata and other metadata related to data provenance. -The following sections describe the :py:class:`~pynwb.base.TimeSeries` and :py:class:`~pynwb.base.ProcessingModules` +The following sections describe the :py:class:`~pynwb.base.TimeSeries` and :py:class:`~pynwb.base.ProcessingModule` classes in further detail. .. _timeseries_overview: @@ -568,7 +568,7 @@ #################### # :py:class:`~hdmf.common.table.DynamicTable` and its subclasses can be converted to a pandas -# :py:class:`~pandas.DataFrame` for convenient analysis using :py:meth:`.DynamicTable.to_dataframe`. +# :py:class:`~pandas.DataFrame` for convenient analysis using :py:meth:`~hdmf.common.table.DynamicTable.to_dataframe`. nwbfile.trials.to_dataframe() diff --git a/docs/gallery/general/plot_timeintervals.py b/docs/gallery/general/plot_timeintervals.py index a04a400c5..4069fd4a4 100644 --- a/docs/gallery/general/plot_timeintervals.py +++ b/docs/gallery/general/plot_timeintervals.py @@ -9,16 +9,14 @@ :py:class:`~pynwb.epoch.TimeIntervals` type. The :py:class:`~pynwb.epoch.TimeIntervals` type is a :py:class:`~hdmf.common.table.DynamicTable` with the following columns: -1. :py:meth:`~pynwb.epoch.TimeIntervals.start_time` and :py:meth:`~pynwb.epoch.TimeIntervals.stop_time` - describe the start and stop times of intervals as floating point offsets in seconds relative to the - :py:meth:`~pynwb.file.NWBFile.timestamps_reference_time` of the file. In addition, -2. :py:class:`~pynwb.epoch.TimeIntervals.tags` is an optional, indexed column used to associate user-defined string - tags with intervals (0 or more tags per time interval) -3. :py:class:`~pynwb.epoch.TimeIntervals.timeseries` is an optional, indexed - :py:class:`~pynwb.base.TimeSeriesReferenceVectorData` column to map intervals directly to ranges in select, - relevant :py:class:`~pynwb.base.TimeSeries` (0 or more per time interval) +1. ``start_time`` and ``stop_time`` describe the start and stop times of intervals as floating point offsets in seconds + relative to the :py:meth:`~pynwb.file.NWBFile.timestamps_reference_time` of the file. In addition, +2. ``tags`` is an optional, indexed column used to associate user-defined string tags with intervals (0 or more tags per + time interval) +3. ``timeseries`` is an optional, indexed :py:class:`~pynwb.base.TimeSeriesReferenceVectorData` column to map intervals + directly to ranges in select, relevant :py:class:`~pynwb.base.TimeSeries` (0 or more per time interval) 4. as a :py:class:`~hdmf.common.table.DynamicTable` user may add additional columns to - :py:meth:`~pynwb.epoch.TimeIntervals` via :py:class:`~hdmf.common.table.DynamicTable.add_column` + :py:meth:`~pynwb.epoch.TimeIntervals` via :py:meth:`~hdmf.common.table.DynamicTable.add_column` .. hint:: :py:meth:`~pynwb.epoch.TimeIntervals` is intended for storing general annotations of time ranges. @@ -84,12 +82,10 @@ # ^^^^^^ # # Trials can be added to an NWB file using the methods :py:meth:`~pynwb.file.NWBFile.add_trial` -# By default, NWBFile only requires trial :py:meth:`~pynwb.file.NWBFile.add_trial.start_time` -# and :py:meth:`~pynwb.file.NWBFile.add_trial.end_time`. The :py:meth:`~pynwb.file.NWBFile.add_trial.tags` -# and :py:meth:`~pynwb.file.NWBFile.add_trial.timeseries` are optional. For -# :py:meth:`~pynwb.file.NWBFile.add_trial.timeseries` we only need to supply the :py:class:`~pynwb.base.TimeSeries`. +# By default, NWBFile only requires trial ``start_time`` and ``stop_time``. The ``tags`` and ``timeseries`` are +# optional. For ``timeseries`` we only need to supply the :py:class:`~pynwb.base.TimeSeries`. # PyNWB automatically calculates the corresponding index range (described by ``idx_start`` and ``count``) for -# the supplied :py:class:`~pynwb.base.TimeSeries based on the given ``start_time`` and ``stop_time`` and +# the supplied :py:class:`~pynwb.base.TimeSeries` based on the given ``start_time`` and ``stop_time`` and # the :py:meth:`~pynwb.base.TimeSeries.timestamps` (or :py:class:`~pynwb.base.TimeSeries.starting_time` # and :py:meth:`~pynwb.base.TimeSeries.rate`) of the given :py:class:`~pynwb.base.TimeSeries`. # @@ -199,7 +195,7 @@ # # To define custom, experiment-specific :py:class:`~pynwb.epoch.TimeIntervals` we can add them # either: 1) when creating the :py:class:`~pynwb.file.NWBFile` by defining the -# :py:meth:`~pynwb.file.NWBFile.__init__.intervals` constructor argument or 2) via the +# ``intervals`` constructor argument or 2) via the # :py:meth:`~pynwb.file.NWBFile.add_time_intervals` or :py:meth:`~pynwb.file.NWBFile.create_time_intervals` # after the :py:class:`~pynwb.file.NWBFile` has been created. # @@ -286,9 +282,9 @@ # Adding TimeSeries references to other tables # -------------------------------------------- # -# Since :py:class:`~pynwb.base.TimeSeriesReferenceVectorData` is a regular :py:class:`~hdmf.common.table.VectoData` +# Since :py:class:`~pynwb.base.TimeSeriesReferenceVectorData` is a regular :py:class:`~hdmf.common.table.VectorData` # type, we can use it to add references to intervals in :py:class:`~pynwb.base.TimeSeries` to any -# :py:class:`~hdmf.common.table.DynamicTable`. In the :py:class:`~pynwb.icephys.IntracellularRecordingTable`, e.g., +# :py:class:`~hdmf.common.table.DynamicTable`. In the :py:class:`~pynwb.icephys.IntracellularRecordingsTable`, e.g., # it is used to reference the recording of the stimulus and response associated with a particular intracellular # electrophysiology recording. # diff --git a/docs/make.bat b/docs/make.bat index dcafe003d..0db0fd778 100644 --- a/docs/make.bat +++ b/docs/make.bat @@ -183,7 +183,7 @@ if "%1" == "changes" ( ) if "%1" == "linkcheck" ( - %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck + %SPHINXBUILD% -W -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck if errorlevel 1 exit /b 1 echo. echo.Link check complete; look for any errors in the above output ^ diff --git a/docs/source/conf.py b/docs/source/conf.py index 5725bd816..c46d2edd5 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -148,6 +148,7 @@ def __call__(self, filename): 'fsspec': ("https://filesystem-spec.readthedocs.io/en/latest/", None), 'nwbwidgets': ("https://nwb-widgets.readthedocs.io/en/latest/", None), 'nwb-overview': ("https://nwb-overview.readthedocs.io/en/latest/", None), + 'zarr': ("https://zarr.readthedocs.io/en/stable/", None), 'hdmf-zarr': ("https://hdmf-zarr.readthedocs.io/en/latest/", None), 'numcodecs': ("https://numcodecs.readthedocs.io/en/latest/", None), } @@ -164,6 +165,10 @@ def __call__(self, filename): 'hdmf-zarr': ('https://hdmf-zarr.readthedocs.io/en/latest/%s', '%s'), } +nitpicky = True +nitpick_ignore = [('py:class', 'Intracomm'), + ('py:class', 'BaseStorageSpec')] + # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/environment-ros3.yml b/environment-ros3.yml index c84b4c090..f5edea6ad 100644 --- a/environment-ros3.yml +++ b/environment-ros3.yml @@ -6,7 +6,7 @@ channels: dependencies: - python==3.11 - h5py==3.8.0 - - hdmf==3.5.4 + - hdmf==3.12.1 - matplotlib==3.7.1 - numpy==1.24.2 - pandas==2.0.0 diff --git a/requirements-min.txt b/requirements-min.txt index 816d53d43..098aea15d 100644 --- a/requirements-min.txt +++ b/requirements-min.txt @@ -1,6 +1,6 @@ # minimum versions of package dependencies for installing PyNWB h5py==2.10 # support for selection of datasets with list of indices added in 2.10 -hdmf==3.12.0 +hdmf==3.12.1 numpy==1.18 pandas==1.1.5 python-dateutil==2.7.3 diff --git a/requirements.txt b/requirements.txt index d09ec7425..0add9c54d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ # pinned dependencies to reproduce an entire development environment to use PyNWB h5py==3.10.0 -hdmf==3.12.0 +hdmf==3.12.1 numpy==1.26.1 pandas==2.1.2 python-dateutil==2.8.2 diff --git a/setup.py b/setup.py index 0e48c269a..d03688905 100755 --- a/setup.py +++ b/setup.py @@ -20,7 +20,7 @@ reqs = [ 'h5py>=2.10', - 'hdmf>=3.12.0', + 'hdmf>=3.12.1', 'numpy>=1.16', 'pandas>=1.1.5', 'python-dateutil>=2.7.3', diff --git a/src/pynwb/__init__.py b/src/pynwb/__init__.py index 5d9bbc57b..200e80074 100644 --- a/src/pynwb/__init__.py +++ b/src/pynwb/__init__.py @@ -147,15 +147,19 @@ def _dec(cls): _dec(container_cls) -def get_nwbfile_version(h5py_file: h5py.File): +@docval({'name': 'h5py_file', 'type': h5py.File, 'doc': 'An NWB file'}, rtype=tuple, + is_method=False,) +def get_nwbfile_version(**kwargs): """ Get the NWB version of the file if it is an NWB file. - :returns: Tuple consisting of: 1) the original version string as stored in the file and - 2) a tuple with the parsed components of the version string, consisting of integers - and strings, e.g., (2, 5, 1, beta). (None, None) will be returned if the file is not a valid NWB file - or the nwb_version is missing, e.g., in the case when no data has been written to the file yet. + + :Returns: Tuple consisting of: 1) the + original version string as stored in the file and 2) a tuple with the parsed components of the version string, + consisting of integers and strings, e.g., (2, 5, 1, beta). (None, None) will be returned if the file is not a + valid NWB file or the nwb_version is missing, e.g., in the case when no data has been written to the file yet. """ # Get the version string for the NWB file + h5py_file = getargs('h5py_file', kwargs) try: nwb_version_string = h5py_file.attrs['nwb_version'] # KeyError occurs when the file is empty (e.g., when creating a new file nothing has been written) @@ -251,7 +255,7 @@ def can_read(path: str): 'doc': 'a path to a namespace, a TypeMap, or a list consisting paths to namespaces and TypeMaps', 'default': None}, {'name': 'file', 'type': [h5py.File, 'S3File'], 'doc': 'a pre-existing h5py.File object', 'default': None}, - {'name': 'comm', 'type': "Intracomm", 'doc': 'the MPI communicator to use for parallel I/O', + {'name': 'comm', 'type': 'Intracomm', 'doc': 'the MPI communicator to use for parallel I/O', 'default': None}, {'name': 'driver', 'type': str, 'doc': 'driver for h5py to use when opening HDF5 file', 'default': None}, {'name': 'herd_path', 'type': str, 'doc': 'The path to the HERD', @@ -327,7 +331,8 @@ def read(self, **kwargs): {'name': 'nwbfile', 'type': 'NWBFile', 'doc': 'the NWBFile object to export. If None, then the entire contents of src_io will be exported', 'default': None}, - {'name': 'write_args', 'type': dict, 'doc': 'arguments to pass to :py:meth:`write_builder`', + {'name': 'write_args', 'type': dict, + 'doc': 'arguments to pass to :py:meth:`~hdmf.backends.io.HDMFIO.write_builder`', 'default': None}) def export(self, **kwargs): """ diff --git a/src/pynwb/base.py b/src/pynwb/base.py index 02d2e3c0f..2197af65f 100644 --- a/src/pynwb/base.py +++ b/src/pynwb/base.py @@ -534,7 +534,7 @@ class TimeSeriesReferenceVectorData(VectorData): then this indicates an invalid link (in practice both ``idx_start`` and ``count`` must always either both be positive or both be negative). When selecting data via the :py:meth:`~pynwb.base.TimeSeriesReferenceVectorData.get` or - :py:meth:`~pynwb.base. TimeSeriesReferenceVectorData.__getitem__` + :py:meth:`~object.__getitem__` functions, ``(-1, -1, TimeSeries)`` values are replaced by the corresponding :py:class:`~pynwb.base.TimeSeriesReferenceVectorData.TIME_SERIES_REFERENCE_NONE_TYPE` tuple to avoid exposing NWB storage internals to the user and simplifying the use of and checking @@ -547,11 +547,11 @@ class TimeSeriesReferenceVectorData(VectorData): TIME_SERIES_REFERENCE_TUPLE = TimeSeriesReference """Return type when calling :py:meth:`~pynwb.base.TimeSeriesReferenceVectorData.get` or - :py:meth:`~pynwb.base. TimeSeriesReferenceVectorData.__getitem__`.""" + :py:meth:`~object.__getitem__`""" TIME_SERIES_REFERENCE_NONE_TYPE = TIME_SERIES_REFERENCE_TUPLE(None, None, None) """Tuple used to represent None values when calling :py:meth:`~pynwb.base.TimeSeriesReferenceVectorData.get` or - :py:meth:`~pynwb.base. TimeSeriesReferenceVectorData.__getitem__`. See also + :py:meth:`~object.__getitem__`. See also :py:class:`~pynwb.base.TimeSeriesReferenceVectorData.TIME_SERIES_REFERENCE_TUPLE`""" @docval({'name': 'name', 'type': str, 'doc': 'the name of this VectorData', 'default': 'timeseries'}, diff --git a/src/pynwb/file.py b/src/pynwb/file.py index 90be96a62..d7a3f31c3 100644 --- a/src/pynwb/file.py +++ b/src/pynwb/file.py @@ -613,7 +613,7 @@ def __check_epochs(self): def add_epoch_column(self, **kwargs): """ Add a column to the epoch table. - See :py:meth:`~pynwb.core.TimeIntervals.add_column` for more details + See :py:meth:`~hdmf.common.table.DynamicTable.add_column` for more details """ self.__check_epochs() self.epoch_tags.update(kwargs.pop('tags', list())) @@ -784,7 +784,7 @@ def add_trial_column(self, **kwargs): def add_trial(self, **kwargs): """ Add a trial to the trial table. - See :py:meth:`~hdmf.common.table.DynamicTable.add_interval` for more details. + See :py:meth:`~pynwb.epoch.TimeIntervals.add_interval` for more details. Required fields are *start_time*, *stop_time*, and any columns that have been added (through calls to `add_trial_columns`). diff --git a/src/pynwb/icephys.py b/src/pynwb/icephys.py index d3a52d12c..3ebb68eaa 100644 --- a/src/pynwb/icephys.py +++ b/src/pynwb/icephys.py @@ -48,7 +48,7 @@ class IntracellularElectrode(NWBContainer): {'name': 'seal', 'type': str, 'doc': 'Information about seal used for recording.', 'default': None}, {'name': 'location', 'type': str, 'doc': 'Area, layer, comments on estimation, stereotaxis coordinates (if in vivo, etc).', 'default': None}, - {'name': 'resistance', 'type': str, 'doc': 'Electrode resistance, unit: Ohm.', 'default': None}, + {'name': 'resistance', 'type': str, 'doc': 'Electrode resistance, unit - Ohm.', 'default': None}, {'name': 'filtering', 'type': str, 'doc': 'Electrode specific filtering.', 'default': None}, {'name': 'initial_access_resistance', 'type': str, 'doc': 'Initial access resistance.', 'default': None}, {'name': 'cell_id', 'type': str, 'doc': 'Unique ID of cell.', 'default': None} @@ -164,11 +164,11 @@ class CurrentClampSeries(PatchClampSeries): 'capacitance_compensation') @docval(*get_docval(PatchClampSeries.__init__, 'name', 'data', 'electrode'), # required - {'name': 'gain', 'type': float, 'doc': 'Units: Volt/Volt'}, + {'name': 'gain', 'type': float, 'doc': 'Units - Volt/Volt'}, *get_docval(PatchClampSeries.__init__, 'stimulus_description'), - {'name': 'bias_current', 'type': float, 'doc': 'Unit: Amp', 'default': None}, - {'name': 'bridge_balance', 'type': float, 'doc': 'Unit: Ohm', 'default': None}, - {'name': 'capacitance_compensation', 'type': float, 'doc': 'Unit: Farad', 'default': None}, + {'name': 'bias_current', 'type': float, 'doc': 'Unit - Amp', 'default': None}, + {'name': 'bridge_balance', 'type': float, 'doc': 'Unit - Ohm', 'default': None}, + {'name': 'capacitance_compensation', 'type': float, 'doc': 'Unit - Farad', 'default': None}, *get_docval(PatchClampSeries.__init__, 'resolution', 'conversion', 'timestamps', 'starting_time', 'rate', 'comments', 'description', 'control', 'control_description', 'sweep_number', 'offset'), {'name': 'unit', 'type': str, 'doc': "The base unit of measurement (must be 'volts')", @@ -267,15 +267,15 @@ class VoltageClampSeries(PatchClampSeries): 'whole_cell_series_resistance_comp') @docval(*get_docval(PatchClampSeries.__init__, 'name', 'data', 'electrode'), # required - {'name': 'gain', 'type': float, 'doc': 'Units: Volt/Amp'}, # required + {'name': 'gain', 'type': float, 'doc': 'Units - Volt/Amp'}, # required *get_docval(PatchClampSeries.__init__, 'stimulus_description'), - {'name': 'capacitance_fast', 'type': float, 'doc': 'Unit: Farad', 'default': None}, - {'name': 'capacitance_slow', 'type': float, 'doc': 'Unit: Farad', 'default': None}, - {'name': 'resistance_comp_bandwidth', 'type': float, 'doc': 'Unit: Hz', 'default': None}, - {'name': 'resistance_comp_correction', 'type': float, 'doc': 'Unit: percent', 'default': None}, - {'name': 'resistance_comp_prediction', 'type': float, 'doc': 'Unit: percent', 'default': None}, - {'name': 'whole_cell_capacitance_comp', 'type': float, 'doc': 'Unit: Farad', 'default': None}, - {'name': 'whole_cell_series_resistance_comp', 'type': float, 'doc': 'Unit: Ohm', 'default': None}, + {'name': 'capacitance_fast', 'type': float, 'doc': 'Unit - Farad', 'default': None}, + {'name': 'capacitance_slow', 'type': float, 'doc': 'Unit - Farad', 'default': None}, + {'name': 'resistance_comp_bandwidth', 'type': float, 'doc': 'Unit - Hz', 'default': None}, + {'name': 'resistance_comp_correction', 'type': float, 'doc': 'Unit - percent', 'default': None}, + {'name': 'resistance_comp_prediction', 'type': float, 'doc': 'Unit - percent', 'default': None}, + {'name': 'whole_cell_capacitance_comp', 'type': float, 'doc': 'Unit - Farad', 'default': None}, + {'name': 'whole_cell_series_resistance_comp', 'type': float, 'doc': 'Unit - Ohm', 'default': None}, *get_docval(PatchClampSeries.__init__, 'resolution', 'conversion', 'timestamps', 'starting_time', 'rate', 'comments', 'description', 'control', 'control_description', 'sweep_number', 'offset'), {'name': 'unit', 'type': str, 'doc': "The base unit of measurement (must be 'amperes')", diff --git a/src/pynwb/image.py b/src/pynwb/image.py index 0295183b8..518ec8a8c 100644 --- a/src/pynwb/image.py +++ b/src/pynwb/image.py @@ -46,7 +46,7 @@ class ImageSeries(TimeSeries): 'is specified. If unit (and data) are not specified, then unit will be set to "unknown".'), 'default': None}, {'name': 'format', 'type': str, - 'doc': 'Format of image. Three types: 1) Image format; tiff, png, jpg, etc. 2) external 3) raw.', + 'doc': 'Format of image. Three types - 1) Image format; tiff, png, jpg, etc. 2) external 3) raw.', 'default': None}, {'name': 'external_file', 'type': ('array_data', 'data'), 'doc': 'Path or URL to one or more external file(s). Field only present if format=external. ' diff --git a/src/pynwb/io/base.py b/src/pynwb/io/base.py index 5b5aac48b..db9c259ef 100644 --- a/src/pynwb/io/base.py +++ b/src/pynwb/io/base.py @@ -73,6 +73,19 @@ def timestamps_carg(self, builder, manager): else: return tstamps_builder.data + @NWBContainerMapper.object_attr("data") + def data_attr(self, container, manager): + ret = container.fields.get('data') + if isinstance(ret, TimeSeries): + owner = ret + curr = owner.fields.get('data') + while isinstance(curr, TimeSeries): + owner = curr + curr = owner.fields.get('data') + data_builder = manager.build(owner) + ret = LinkBuilder(data_builder['data'], 'data') + return ret + @NWBContainerMapper.constructor_arg("data") def data_carg(self, builder, manager): # handle case where a TimeSeries is read and missing data @@ -105,7 +118,10 @@ def unit_carg(self, builder, manager): data_builder = manager.construct(target.parent) else: data_builder = target - unit_value = data_builder.attributes.get('unit') + if isinstance(data_builder, TimeSeries): # Data linked in another timeseries + unit_value = data_builder.unit + else: # DatasetBuilder owned by this timeseries + unit_value = data_builder.attributes.get('unit') if unit_value is None: return timeseries_cls.DEFAULT_UNIT return unit_value diff --git a/src/pynwb/io/utils.py b/src/pynwb/io/utils.py index 23b41b6b4..e607b9589 100644 --- a/src/pynwb/io/utils.py +++ b/src/pynwb/io/utils.py @@ -17,7 +17,7 @@ def get_nwb_version(builder: Builder, include_prerelease=False) -> Tuple[int, .. if include_prerelease=True, (2, 0, 0, "b") is returned; else, (2, 0, 0) is returned. :param builder: Any builder within an NWB file. - :type builder: Builder + :type builder: :py:class:`~hdmf.build.builders.Builder` :param include_prerelease: Whether to include prerelease information in the returned tuple. :type include_prerelease: bool :return: The version of the NWB file, as a tuple. diff --git a/src/pynwb/misc.py b/src/pynwb/misc.py index 4d977b4f2..14c2e08d1 100644 --- a/src/pynwb/misc.py +++ b/src/pynwb/misc.py @@ -265,7 +265,7 @@ class DecompositionSeries(TimeSeries): 'shape': (None, None, None)}, *get_docval(TimeSeries.__init__, 'description'), {'name': 'metric', 'type': str, # required - 'doc': "metric of analysis. recommended: 'phase', 'amplitude', 'power'"}, + 'doc': "metric of analysis. recommended - 'phase', 'amplitude', 'power'"}, {'name': 'unit', 'type': str, 'doc': 'SI unit of measurement', 'default': 'no unit'}, {'name': 'bands', 'type': DynamicTable, 'doc': 'a table for describing the frequency bands that the signal was decomposed into', 'default': None}, diff --git a/src/pynwb/ophys.py b/src/pynwb/ophys.py index e3d9f8f6d..bdff47592 100644 --- a/src/pynwb/ophys.py +++ b/src/pynwb/ophys.py @@ -57,15 +57,15 @@ class ImagingPlane(NWBContainer): 'doc': 'Rate images are acquired, in Hz. If the corresponding TimeSeries is present, the rate should be ' 'stored there instead.', 'default': None}, {'name': 'manifold', 'type': 'array_data', - 'doc': ('DEPRECATED: Physical position of each pixel. size=("height", "width", "xyz"). ' + 'doc': ('DEPRECATED - Physical position of each pixel. size=("height", "width", "xyz"). ' 'Deprecated in favor of origin_coords and grid_spacing.'), 'default': None}, {'name': 'conversion', 'type': float, - 'doc': ('DEPRECATED: Multiplier to get from stored values to specified unit (e.g., 1e-3 for millimeters) ' + 'doc': ('DEPRECATED - Multiplier to get from stored values to specified unit (e.g., 1e-3 for millimeters) ' 'Deprecated in favor of origin_coords and grid_spacing.'), 'default': 1.0}, {'name': 'unit', 'type': str, - 'doc': 'DEPRECATED: Base unit that coordinates are stored in (e.g., Meters). ' + 'doc': 'DEPRECATED - Base unit that coordinates are stored in (e.g., Meters). ' 'Deprecated in favor of origin_coords_unit and grid_spacing_unit.', 'default': 'meters'}, {'name': 'reference_frame', 'type': str, diff --git a/src/pynwb/retinotopy.py b/src/pynwb/retinotopy.py index 1dc0f365d..b610ca37f 100644 --- a/src/pynwb/retinotopy.py +++ b/src/pynwb/retinotopy.py @@ -129,9 +129,9 @@ # 'Description should be something like ["altitude", "azimuth"] or ["radius", "theta"].'}, # {'name': 'focal_depth_image', 'type': FocalDepthImage, # 'doc': 'Gray-scale image taken with same settings/parameters (e.g., focal depth, wavelength) ' -# 'as data collection. Array format: [rows][columns].'}, +# 'as data collection. Array format - [rows][columns].'}, # {'name': 'vasculature_image', 'type': RetinotopyImage, -# 'doc': 'Gray-scale anatomical image of cortical surface. Array structure: [rows][columns].'}, +# 'doc': 'Gray-scale anatomical image of cortical surface. Array structure - [rows][columns].'}, # {'name': 'name', 'type': str, 'doc': 'the name of this container', 'default': 'ImagingRetinotopy'}) # def __init__(self, **kwargs): # axis_1_phase_map, axis_1_power_map, axis_2_phase_map, axis_2_power_map, axis_descriptions, \ @@ -139,6 +139,8 @@ # 'axis_1_phase_map', 'axis_1_power_map', 'axis_2_phase_map', 'axis_2_power_map', # 'axis_descriptions', 'focal_depth_image', 'sign_map', 'vasculature_image', kwargs) # super().__init__(**kwargs) +# warnings.warn("The ImagingRetinotopy class currently cannot be written to or read from a file. " +# "This is a known bug and will be fixed in a future release of PyNWB.") # self.axis_1_phase_map = axis_1_phase_map # self.axis_1_power_map = axis_1_power_map # self.axis_2_phase_map = axis_2_phase_map diff --git a/src/pynwb/spec.py b/src/pynwb/spec.py index 94271118a..fe97b6eae 100644 --- a/src/pynwb/spec.py +++ b/src/pynwb/spec.py @@ -62,7 +62,7 @@ def neurodata_type_inc(self): class BaseStorageOverride: ''' This class is used for the purpose of overriding - BaseStorageSpec classmethods, without creating diamond + :py:class:`~hdmf.spec.spec.BaseStorageSpec` classmethods, without creating diamond inheritance hierarchies. ''' diff --git a/src/pynwb/testing/icephys_testutils.py b/src/pynwb/testing/icephys_testutils.py index 732f312e6..3de4619d4 100644 --- a/src/pynwb/testing/icephys_testutils.py +++ b/src/pynwb/testing/icephys_testutils.py @@ -60,10 +60,10 @@ def create_icephys_testfile(filename=None, add_custom_columns=True, randomize_da :param randomize_data: Randomize data values in the stimulus and response :type randomize_data: bool - :returns: ICEphysFile NWBFile object created for writing. NOTE: If filename is provided then + :returns: NWBFile object with icephys data created for writing. NOTE: If filename is provided then the file is written to disk, but the function does not read the file back. If you want to use the file from disk then you will need to read it with NWBHDF5IO. - :rtype: ICEphysFile + :rtype: NWBFile """ nwbfile = NWBFile( session_description='my first synthetic recording', diff --git a/src/pynwb/testing/testh5io.py b/src/pynwb/testing/testh5io.py index b45407bfb..c7b3bfdcc 100644 --- a/src/pynwb/testing/testh5io.py +++ b/src/pynwb/testing/testh5io.py @@ -79,7 +79,7 @@ def test_roundtrip_export(self): self.assertIs(self.read_exported_nwbfile.objects[self.container.object_id], self.read_container) self.assertContainerEqual(self.read_container, self.container, ignore_hdmf_attrs=True) - def roundtripContainer(self, cache_spec=False): + def roundtripContainer(self, cache_spec=True): """Add the Container to an NWBFile, write it to file, read the file, and return the Container from the file. """ session_description = 'a file to test writing and reading a %s' % self.container_type @@ -116,7 +116,7 @@ def roundtripContainer(self, cache_spec=False): self.reader = None raise e - def roundtripExportContainer(self, cache_spec=False): + def roundtripExportContainer(self, cache_spec=True): """ Add the test Container to an NWBFile, write it to file, read the file, export the read NWBFile to another file, and return the test Container from the file @@ -163,18 +163,14 @@ def getContainer(self, nwbfile): def validate(self): """ Validate the created files """ if os.path.exists(self.filename): - with NWBHDF5IO(self.filename, mode='r') as io: - errors = pynwb_validate(io) - if errors: - for err in errors: - raise Exception(err) + errors, _ = pynwb_validate(paths=[self.filename]) + if errors: + raise Exception("\n".join(errors)) if os.path.exists(self.export_filename): - with NWBHDF5IO(self.filename, mode='r') as io: - errors = pynwb_validate(io) - if errors: - for err in errors: - raise Exception(err) + errors, _ = pynwb_validate(paths=[self.export_filename]) + if errors: + raise Exception("\n".join(errors)) class AcquisitionH5IOMixin(NWBH5IOMixin): @@ -294,7 +290,7 @@ def test_roundtrip_export(self): self.assertIs(self.read_exported_nwbfile.objects[self.container.object_id], self.read_container) self.assertContainerEqual(self.read_container, self.container, ignore_hdmf_attrs=True) - def roundtripContainer(self, cache_spec=False): + def roundtripContainer(self, cache_spec=True): """Write the file, validate the file, read the file, and return the Container from the file. """ @@ -325,7 +321,7 @@ def roundtripContainer(self, cache_spec=False): self.reader = None raise e - def roundtripExportContainer(self, cache_spec=False): + def roundtripExportContainer(self, cache_spec=True): """ Roundtrip the container, then export the read NWBFile to a new file, validate the files, and return the test Container from the file. @@ -366,13 +362,11 @@ def roundtripExportContainer(self, cache_spec=False): def validate(self): """Validate the created files.""" if os.path.exists(self.filename): - with NWBHDF5IO(self.filename, mode='r') as io: - errors = pynwb_validate(io) - if errors: - raise Exception("\n".join(errors)) + errors, _ = pynwb_validate(paths=[self.filename]) + if errors: + raise Exception("\n".join(errors)) if os.path.exists(self.export_filename): - with NWBHDF5IO(self.filename, mode='r') as io: - errors = pynwb_validate(io) - if errors: - raise Exception("\n".join(errors)) + errors, _ = pynwb_validate(paths=[self.export_filename]) + if errors: + raise Exception("\n".join(errors)) diff --git a/tests/integration/hdf5/test_base.py b/tests/integration/hdf5/test_base.py index 1e855f3ce..60f8510ff 100644 --- a/tests/integration/hdf5/test_base.py +++ b/tests/integration/hdf5/test_base.py @@ -46,6 +46,27 @@ def test_timestamps_linking(self): tsb = nwbfile.acquisition['b'] self.assertIs(tsa.timestamps, tsb.timestamps) + def test_data_linking(self): + ''' Test that data get linked to in TimeSeries ''' + tsa = TimeSeries(name='a', data=np.linspace(0, 1, 1000), timestamps=np.arange(1000.), unit='m') + tsb = TimeSeries(name='b', data=tsa, timestamps=np.arange(1000.), unit='m') + tsc = TimeSeries(name='c', data=tsb, timestamps=np.arange(1000.), unit='m') + nwbfile = NWBFile(identifier='foo', + session_start_time=datetime(2017, 5, 1, 12, 0, 0, tzinfo=tzlocal()), + session_description='bar') + nwbfile.add_acquisition(tsa) + nwbfile.add_acquisition(tsb) + nwbfile.add_acquisition(tsc) + with NWBHDF5IO(self.path, 'w') as io: + io.write(nwbfile) + with NWBHDF5IO(self.path, 'r') as io: + nwbfile = io.read() + tsa = nwbfile.acquisition['a'] + tsb = nwbfile.acquisition['b'] + tsc = nwbfile.acquisition['c'] + self.assertIs(tsa.data, tsb.data) + self.assertIs(tsa.data, tsc.data) + class TestImagesIO(AcquisitionH5IOMixin, TestCase): diff --git a/tests/integration/hdf5/test_modular_storage.py b/tests/integration/hdf5/test_modular_storage.py index 6c86fc615..fba5d02db 100644 --- a/tests/integration/hdf5/test_modular_storage.py +++ b/tests/integration/hdf5/test_modular_storage.py @@ -14,29 +14,35 @@ class TestTimeSeriesModular(TestCase): def setUp(self): - self.start_time = datetime(1971, 1, 1, 12, tzinfo=tzutc()) + # File paths + self.data_filename = os.path.join(os.getcwd(), 'test_time_series_modular_data.nwb') + self.link_filename = os.path.join(os.getcwd(), 'test_time_series_modular_link.nwb') + # Make the data container file write + self.start_time = datetime(1971, 1, 1, 12, tzinfo=tzutc()) self.data = np.arange(2000).reshape((1000, 2)) self.timestamps = np.linspace(0, 1, 1000) - + # The container before roundtrip self.container = TimeSeries( name='data_ts', unit='V', data=self.data, timestamps=self.timestamps ) + self.data_read_io = None # HDF5IO used for reading the main data file + self.read_data_nwbfile = None # The NWBFile read after roundtrip + self.read_data_container = None # self.container after roundtrip - self.data_filename = os.path.join(os.getcwd(), 'test_time_series_modular_data.nwb') - self.link_filename = os.path.join(os.getcwd(), 'test_time_series_modular_link.nwb') - - self.read_container = None - self.link_read_io = None - self.data_read_io = None + # Variables for the second file which links the main data file + self.link_container = None # The container with the links before write + self.read_link_container = None # The container with the links after roundtrip + self.read_link_nwbfile = None # The NWBFile container containing link_container after roundtrip + self.link_read_io = None # HDF5IO use for reading the read_link_nwbfile def tearDown(self): - if self.read_container: - self.read_container.data.file.close() - self.read_container.timestamps.file.close() + if self.read_link_container: + self.read_link_container.data.file.close() + self.read_link_container.timestamps.file.close() if self.link_read_io: self.link_read_io.close() if self.data_read_io: @@ -64,49 +70,83 @@ def roundtripContainer(self): data_write_io.write(data_file) # read data file - with HDF5IO(self.data_filename, 'r', manager=get_manager()) as self.data_read_io: - data_file_obt = self.data_read_io.read() - - # write "link file" with timeseries.data that is an external link to the timeseries in "data file" - # also link timeseries.timestamps.data to the timeseries.timestamps in "data file" - with HDF5IO(self.link_filename, 'w', manager=get_manager()) as link_write_io: - link_file = NWBFile( - session_description='a test file', - identifier='link_file', - session_start_time=self.start_time - ) - self.link_container = TimeSeries( - name='test_mod_ts', - unit='V', - data=data_file_obt.get_acquisition('data_ts'), # test direct link - timestamps=H5DataIO( - data=data_file_obt.get_acquisition('data_ts').timestamps, - link_data=True # test with setting link data - ) - ) - link_file.add_acquisition(self.link_container) - link_write_io.write(link_file) + self.data_read_io = HDF5IO(self.data_filename, 'r', manager=get_manager()) + self.read_data_nwbfile = self.data_read_io.read() + self.read_data_container = self.read_data_nwbfile.get_acquisition('data_ts') - # note that self.link_container contains a link to a dataset that is now closed + # write "link file" with timeseries.data that is an external link to the timeseries in "data file" + # also link timeseries.timestamps.data to the timeseries.timestamps in "data file" + with HDF5IO(self.link_filename, 'w', manager=get_manager()) as link_write_io: + link_file = NWBFile( + session_description='a test file', + identifier='link_file', + session_start_time=self.start_time + ) + self.link_container = TimeSeries( + name='test_mod_ts', + unit='V', + data=H5DataIO( + data=self.read_data_container.data, + link_data=True # test with setting link data + ), + timestamps=H5DataIO( + data=self.read_data_container.timestamps, + link_data=True # test with setting link data + ) + ) + link_file.add_acquisition(self.link_container) + link_write_io.write(link_file) # read the link file self.link_read_io = HDF5IO(self.link_filename, 'r', manager=get_manager()) - self.read_nwbfile = self.link_read_io.read() - return self.getContainer(self.read_nwbfile) + self.read_link_nwbfile = self.link_read_io.read() + return self.getContainer(self.read_link_nwbfile) def test_roundtrip(self): - self.read_container = self.roundtripContainer() - - # make sure we get a completely new object - self.assertIsNotNone(str(self.container)) # added as a test to make sure printing works + # Roundtrip the container + self.read_link_container = self.roundtripContainer() + + # 1. Make sure our containers are set correctly for the test + # 1.1: Make sure the container we read is not identical to the container we used for writing + self.assertNotEqual(id(self.link_container), id(self.read_link_container)) + self.assertNotEqual(id(self.container), id(self.read_data_container)) + # 1.2: Make sure the container we read is indeed the correct container we should use for testing + self.assertIs(self.read_link_nwbfile.objects[self.link_container.object_id], self.read_link_container) + self.assertIs(self.read_data_nwbfile.objects[self.container.object_id], self.read_data_container) + # 1.3: Make sure the object_ids of the container we wrote and read are the same + self.assertEqual(self.read_link_container.object_id, self.link_container.object_id) + self.assertEqual(self.read_data_container.object_id, self.container.object_id) + # 1.4: Make sure the object_ids between the source data and link data container are different + self.assertNotEqual(self.read_link_container.object_id, self.read_data_container.object_id) + + # Test that printing works for the source data and linked data container + self.assertIsNotNone(str(self.container)) + self.assertIsNotNone(str(self.read_data_container)) self.assertIsNotNone(str(self.link_container)) - self.assertIsNotNone(str(self.read_container)) - self.assertFalse(self.link_container.timestamps.valid) - self.assertTrue(self.read_container.timestamps.id.valid) - self.assertNotEqual(id(self.link_container), id(self.read_container)) - self.assertIs(self.read_nwbfile.objects[self.link_container.object_id], self.read_container) - self.assertContainerEqual(self.read_container, self.container, ignore_name=True, ignore_hdmf_attrs=True) - self.assertEqual(self.read_container.object_id, self.link_container.object_id) + self.assertIsNotNone(str(self.read_link_container)) + + # Test that timestamps and data are valid after write + self.assertTrue(self.read_link_container.timestamps.id.valid) + self.assertTrue(self.read_link_container.data.id.valid) + self.assertTrue(self.read_data_container.timestamps.id.valid) + self.assertTrue(self.read_data_container.data.id.valid) + + # Make sure the data in the read data container and linked data container match the original container + self.assertContainerEqual(self.read_link_container, self.container, ignore_name=True, ignore_hdmf_attrs=True) + self.assertContainerEqual(self.read_data_container, self.container, ignore_name=True, ignore_hdmf_attrs=True) + + # Make sure the timestamps and data are linked correctly. I.e., the filename of the h5py dataset should + # match between the data file and the file with links even-though they have been read from different files + self.assertEqual( + self.read_data_container.data.file.filename, # Path where the source data is stored + self.read_link_container.data.file.filename # Path where the linked h5py dataset points to + ) + self.assertEqual( + self.read_data_container.timestamps.file.filename, # Path where the source data is stored + self.read_link_container.timestamps.file.filename # Path where the linked h5py dataset points to + ) + + # validate both the source data and linked data file via the pynwb validator self.validate() def test_link_root(self): @@ -159,3 +199,55 @@ def validate(self): def getContainer(self, nwbfile): return nwbfile.get_acquisition('test_mod_ts') + + +class TestTimeSeriesModularLinkViaTimeSeries(TestTimeSeriesModular): + """ + Same as TestTimeSeriesModular but creating links by setting TimeSeries.data + and TimeSeries.timestamps to the other TimeSeries on construction, rather than + using H5DataIO. + """ + def setUp(self): + super().setUp() + self.skipTest("This behavior is currently broken. See issue .") + + def roundtripContainer(self): + # create and write data file + data_file = NWBFile( + session_description='a test file', + identifier='data_file', + session_start_time=self.start_time + ) + data_file.add_acquisition(self.container) + + with HDF5IO(self.data_filename, 'w', manager=get_manager()) as data_write_io: + data_write_io.write(data_file) + + # read data file + self.data_read_io = HDF5IO(self.data_filename, 'r', manager=get_manager()) + self.read_data_nwbfile = self.data_read_io.read() + self.read_data_container = self.read_data_nwbfile.get_acquisition('data_ts') + + # write "link file" with timeseries.data that is an external link to the timeseries in "data file" + # also link timeseries.timestamps.data to the timeseries.timestamps in "data file" + with HDF5IO(self.link_filename, 'w', manager=get_manager()) as link_write_io: + link_file = NWBFile( + session_description='a test file', + identifier='link_file', + session_start_time=self.start_time + ) + self.link_container = TimeSeries( + name='test_mod_ts', + unit='V', + data=self.read_data_container, # <--- This is the main difference to TestTimeSeriesModular + timestamps=self.read_data_container # <--- This is the main difference to TestTimeSeriesModular + ) + link_file.add_acquisition(self.link_container) + link_write_io.write(link_file) + + # note that self.link_container contains a link to a dataset that is now closed + + # read the link file + self.link_read_io = HDF5IO(self.link_filename, 'r', manager=get_manager()) + self.read_link_nwbfile = self.link_read_io.read() + return self.getContainer(self.read_link_nwbfile)