Streaming add remfile

CHANGELOG.md

@@ -6,6 +6,9 @@
 - Add `NWBHDF5IO.can_read()`. @bendichter [#1703](https://github.com/NeurodataWithoutBorders/pynwb/pull/1703)
 - Add `pynwb.get_nwbfile_version()`. @bendichter [#1703](https://github.com/NeurodataWithoutBorders/pynwb/pull/1703)
 
+### Documentation and tutorial enhancements
+- Add RemFile to streaming tutorial @bendichter [#1761](https://github.com/NeurodataWithoutBorders/pynwb/pull/1761)
+
 ## PyNWB 2.5.0 (August 18, 2023)
 
 ### Enhancements and minor changes

docs/gallery/advanced_io/streaming.py

@@ -122,6 +122,30 @@
 #        conda install -c conda-forge "h5py>=3.2"
 
 
+##################################################
+# Method 3: remfile
+# -----------------
+# remfile is another library that enables indexing and streaming of files in s3. remfile is simple, fast, and allows for
+# caching of data in the local filesystem. The one caveat of remfile is that it is a very new project that has not been
+# tested in a variety of use-cases. You can install remfile with pip:
+#
+# .. code-block:: bash
+#
+#   pip install remfile
+#
+# Then remfile can be used similarly to fsspec
+
+import h5py
+from pynwb import NWBHDF5IO
+import remfile
+
+rem_file = remfile.File(s3_url)
+
+with h5py.File(rem_file, "r") as h5py_file:
+    with NWBHDF5IO(file=h5py_file, load_namespaces=True) as io:
+        nwbfile = io.read()
+        print(nwbfile.acquisition["lick_times"].time_series["lick_left_times"].data[:])
+
 ##################################################
 # Which streaming method to choose?
 # ---------------------------------

environment-ros3.yml

@@ -16,3 +16,6 @@ dependencies:
   - fsspec==2023.6.0
   - requests==2.28.1
   - aiohttp==3.8.3
+  - pip
+  - pip:
+    - remfile==0.1.7