Relevant features of holographic stimulus

[alessandratrapani]

@weiglszonja @pauladkisson @CodyCBakerPhD
This first issue is just to collect notes on relevant features that should be included in the new neurodata type for HolographicSeries

[weiglszonja]

@alessandratrapani Thank you for working on this!
I started collecting the metadata from Pinto holographic stimulation data:

• "laser" (type of laser e.g. "Monaco")
• "power" (power of laser e.g. "400")
• "centroids" (x, y position of the cells, with respect to imaging field of view)
  -"cell group" (e.g. "Group 3" is cell indices from 21-30)
  stimulation pattern:
• "type" (e.g. "spiral")
• "density" (e.g. "0.026") -> called "spiralWidth"
• "size" (e.g. spiral size in microns e.g. "15")

Example output from xml for reference:

<?xml version="1.0" encoding="utf-8"?>
<PVMarkPointSeriesElements Category="multi_cell_stim_30_points" Name="SLM_30_pseudorandom_1" Iterations="1" IterationDelay="0.00" CalcFunctMap="False">
  <PVMarkPointElement Repetitions="5" UncagingLaser="Monaco" UncagingLaserPower="400" TriggerFrequency="None" TriggerSelection="None" TriggerCount="1" AsyncSyncFrequency="None" VoltageOutputCategoryName="None" VoltageRecCategoryName="None" parameterSet="CurrentSettings">
    <PVGalvoPointElement InitialDelay="15000" InterPointDelay="35" Duration="15" SpiralRevolutions="5" AllPointsAtOnce="True" Use3D="False" Points="Group 3" Indices="21-30">
      <Point Index="21" X="0.640033747469953" Y="0.0614025480962288" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="22" X="0.171501640727355" Y="0.104856471848038" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="23" X="0.197842514511006" Y="0.171083853539225" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="24" X="0.557873666021733" Y="0.201207121040656" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="25" X="0.212477234654433" Y="0.425388481242167" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="26" X="0.333489496505596" Y="0.626025708108721" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="27" X="0.690932801161692" Y="0.664648713986135" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="28" X="0.723288999217156" Y="0.71065600700909" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="29" X="0.245011904906769" Y="0.807846286764258" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
      <Point Index="30" X="0.38223065908971" Y="0.880054515143771" IsSpiral="True" SpiralWidth="0.0264456981664316" SpiralHeight="0.0264456981664316" SpiralSizeInMicrons="15" />
...

[CodyCBakerPhD]

For @weiglszonja how do we communicate the spiral activation pattern the laser takes? Is that also typical of the Adesnik data?

[alessandratrapani]

There is no spiral activation pattern in Adesnik data. They target the whole ROI volume at once (in ScanImage coordinates) where they expect to be cell bodies.
I am also collecting the metadata for Adesnik data and will dig into the spiral pattern in the next few days.

[alessandratrapani]

Sorry for taking so long but I wanted to have a clearer picture. Here are some basic notes I gathered from different papers, (most of the notes are from the one suggested by Pinto lab for the methodology they used).
I gather some features that I think are good to have in this extension, I just need a hand in understanding where to place them. As you can notice there is of course a lot of overlap with the ophys (two photon) module and the ogen module. One thing that I am still not sure how to define is the equivalent of data in the pynwb.ogen.OptogeneticSeries.
NB: there is a wide variety of combinations for the holographic optogenetic stimulation method. I am thinking of restricting this extension to the one that uses two-photon techniques, but I am looking for a more expert eye on this trade-off between being as general as possible to include the majority of cases and trying to include all the critical features.

Critical features for holographic optogenetic stimulation

Effector (opsin, membrane channel): (equivalent of Indicator for imaging)

• excitation wavelength
• emission wavelength

Laser: (Device)

• type
• wavelength
• average power
• peak power
• pulse repetition rate
• pulse width

Beam pattern (on single cells): (probably it should go in the equivalent pynwb.ogen.OptogeneticStimulusSite)
for 2-photon stimulation, there are two main categories: serial (e.g. raster scanning, spiral scanning) and parallel (e.g. temporal focusing, digital holography). Each method has its own parameters, for instance:
spiral scanning:  (as in Pinto holographic stimulation data, reference paper for the methodology)

• spiral duration
• repetition frequency
• revolution

temporal focusing: (as in Abdeladim, Lamiae, et al. "Probing inter-areal computations with a cellular resolution two-photon holographic mesoscope." bioRxiv (2023): 2023-03.)

• Optical lateral point-spread-functions
• Optical axial point-spread-functions

ROI excitation (equivalent of ROIresponse for imaging)

• rois (DynamicTableRegion) – a table region corresponding to the ROIs that were stimulated

[weiglszonja]

@alessandratrapani Thank you for collecting these features!
I'm trying to map out what we will add in this extension (based it off from All-optical interrogation of neural circuits in behaving mice):

1. PhotoStimulationLaser (extends Device)
   (with the metadata you collected)
2. StimulationTarget (extends ImagingPlane)

• indicator (the activity indicator the neuron's expresses)
• effector (opsin, membrane channel)
• field of view (FOV) : the imaging field of view where the photo stimulation laser is targeted
• links to PhotoStimulationLaser

3. HolographicStimulusPattern (extends pynwb.ogen.OptogeneticStimulusSite)
   (with the metadata you collected)
4. HolographicSeries (extends pynwb.ogen.OptogeneticSeries)

• links to HolographicStimulusPattern
• links to rois (DynamicTableRegion)
• data: I'm not sure about this either... the times when the cells were stimulated ? (ntimes x nrois)

tool mentioned in the article for photoactivation response mapping:
https://github.com/llerussell/Naparm

[alessandratrapani]

• data: I'm not sure about this either... the times when the cells were stimulated? (ntimes x nrois)

I thought this too, I just don't know if we have the actual TimeSeries for the stimulus to each ROI

[alessandratrapani]

3. HolographicStimulusPattern (extends pynwb.ogen.OptogeneticStimulusSite)
(with the metadata you collected)

How can we deal with different stimulus patterns with different descriptive parameters?

[weiglszonja]

How can we deal with different stimulus patterns with different descriptive parameters?

How about having "base" HolographicStimulusPattern with name and description and the descriptive parameters are added as DynamicTable for flexibility?
Or creating a pre-defined stimulus pattern for each we know (e.g. first one would be the spiral type with the metadata from the articles)

[weiglszonja]

• data: I'm not sure about this either... the times when the cells were stimulated? (ntimes x nrois)

I thought this too, I just don't know if we have the actual TimeSeries for the stimulus to each ROI

I'm compiling the MarkPoints.xml files to collect the times based on Neto's description

<PVGalvoPointElement InitialDelay="15000" InterPointDelay="35" Duration="15" SpiralRevolutions="5" AllPointsAtOnce="True" Use3D="False" Points="Group 3" Indices="21-30">

• "relative_start_time" : InitialDelay converted to seconds
• "relative_end_time": number of spiral repetitions * (delay time between spirals + duration of each spiral) + previous relative start time

This was my understanding but could be wrong.

[CodyCBakerPhD]

This looks like it's really starting to come together! Very cool to finally be able to dig into specific details of how this method works...

Laser: (Device)

type
wavelength
average power
peak power
pulse repetition rate
pulse width

As discussed from our meeting; I highly recommend avoiding having multiple attributes that are interrelated functionally - in this case 'average power' being a function of other attributes - meaning it's possible for someone to input inconsistent values across the neurodata object (BTW can you post a screenshot or copy/paste of that section of the paper you showed me to have it here for reference?)

If it's unavoidable it means we'd which we then need special validators (Inspector checks) for it, but if there's a better design to be created here that can avoid that complexity, it would be preferred

Also some of these attributes you mentioned being specific only to pulse lasers - if it's possible to condense this list down to more generic attributes that can apply to all laser types? Similar to the goal of the MicroscopySeries on the other extension?

[CodyCBakerPhD]

Maybe crazy idea for super generic representation of the stimulus patterns, since they have such diversity that text based metadata alone might never capture it; how about providing a 'summary image' of sorts that visually describes the excitation pattern? On the technological side we could have helper functions for retrieving the built-in types (all 4 of those from the paper figure) given a users string specification, but this would also enable them to specify anything customized in a way that is more explicit

[alessandratrapani]

how about providing a 'summary image' of sorts that visually describes the excitation pattern?

I think it can be feasible to describe the shape of the stimulus on the single ROIs, as in the case of the spiral pattern, but if the shape of the stimulus varies in time for the single ROI or it's different for each ROI, I think it scales up to a level of complexity that it's not necessary in the NWB representation. We don't report all the parameters in the preprocessing algorithms, as well as all the details in the surgical procedures, and I think this is on the same level since it's the description of the experimental protocol. Do you agree?

Also some of these attributes you mentioned being specific only to pulse lasers - if it's possible to condense this list down to more generic attributes that can apply to all laser types?

After reasoning on this I think that the way to go is to report only the laser power as a time series (like in the ogen module) for each targeted ROIs, similar to the ROIresponse container, but instead of the indicator we would have the effector and the default unit would be W. Does it make sense to you?

[alessandratrapani]

I would like to propose this new structure in light of the new considerations:

The new parameters added are marked in bold

HolographicStimulusPattern

to be discussed. Proposals:

1. 'summary image' that visually describes the excitation pattern of a single ROI
2. name, description and a DynamicTable with the descriptive parameters
3. 4D image - hologram

HolographicStimulationTarget

Bases: ImagingPlane
Parameters:

• name (str) – the name of this container
• optical_channel (list or OpticalChannel) – One of possibly many groups storing channel-specific data.
• description (str) – Description of this ImagingPlane.
• device (Device) – the device that was used to record
• excitation_lambda (float) – Excitation wavelength in nm.
• indicator (str) – Calcium indicator
• effector (str) - Opsin or channel membrane expressed by the targeted cell
• location (str) – Location of the image plane.
• imaging_rate (float) – Rate images are acquired, in Hz. If the corresponding TimeSeries is present, the rate should be stored there instead.
• reference_frame (str) – Describes position and reference frame of manifold based on position of first element in manifold.
• origin_coords (ndarray or list or tuple or Dataset or Array or StrDataset or HDMFDataset or AbstractDataChunkIterator) – Physical location of the first element of the imaging plane (0, 0) for 2-D data or (0, 0, 0) for 3-D data. See also reference_frame for what the physical location is relative to (e.g., bregma).
• origin_coords_unit (str) – Measurement units for origin_coords. The default value is ‘meters’.
• grid_spacing (ndarray or list or tuple or Dataset or Array or StrDataset or HDMFDataset or AbstractDataChunkIterator) – Space between pixels in (x, y) or voxels in (x, y, z) directions, in the specified unit. Assumes imaging plane is a regular grid. See also reference_frame to interpret the grid.
• grid_spacing_unit (str) – Measurement units for grid_spacing. The default value is ‘meters’.

HolographicStimulusSite

Bases: OptogeneticStimulusSite
Holographic optogenetic stimulus site.
Parameters:

• name (str) – The name of this stimulus site.
• device (Device) – The device that was used.
• description (str) – Description of site.
• excitation_lambda (float) – Excitation wavelength in nm.
• location (str) – Location of stimulation site.
• stimulus_target (HolographicStimulationTarget)
• stimulus_pattern (HolographicStimulusPattern)

HolographicSeries

Bases: OptogeneticSeries
Holographic optogenetic stimulus. The data field is in units of watts.
Parameters:

• name (str) – The name of this TimeSeries dataset
• data (ndarray or list or tuple or Dataset or Array or StrDataset or HDMFDataset or AbstractDataChunkIterator or DataIO or TimeSeries) – The data values. May be 1D or 2D. The first dimension must be time. The optional second dimension represents ROIs
• unit (str) - The base unit of the stimulus (should be SI unit - default W)
• rois (DynamicTableRegion) – a table region corresponding to the ROIs that were stimulated
• site (HolographicStimulusSite) – The site to which this stimulus was applied.
• resolution (float) – The smallest meaningful difference (in specified unit) between values in data
• conversion (float) – Scalar to multiply each element in data to convert it to the specified unit
• timestamps (ndarray or list or tuple or Dataset or Array or StrDataset or HDMFDataset or AbstractDataChunkIterator or DataIO or TimeSeries) – Timestamps for samples stored in data
• starting_time (float) – The timestamp of the first sample
• rate (float) – Sampling rate in Hz
• comments (str) – Human-readable comments about this TimeSeries dataset
• description (str) – Description of this TimeSeries dataset
• control (Iterable) – Numerical labels that apply to each element in data
• control_description (Iterable) – Description of each control value
• offset (float) – Scalar to add to each element in the data scaled by ‘conversion’ to finish converting it to the specified unit.

[weiglszonja]

@alessandratrapani this looks great! For HolographicSeries, didn't you mean RoiResponseSeries or OptogeneticSeries for base?

[alessandratrapani]

@alessandratrapani this looks great! For HolographicSeries, didn't you mean RoiResponseSeries or OptogeneticSeries for base?

Oh yeah, sorry for the confusion. I meant the base should be OptogeneticSeries, but similarly as RoiResponseSeries should have rois and unit (basically they are the same and differ by these parameters and site).

[pauladkisson]

@alessandratrapani Looks really good! Thank you for putting all of this together. I have a few questions/notes:

HolographicStimulationTarget
Bases: ImagingPlane
Parameters:

name (str) – the name of this container
optical_channel (list or OpticalChannel) – One of possibly many groups storing channel-specific data.
description (str) – Description of this ImagingPlane.
device (Device) – the device that was used to record
excitation_lambda (float) – Excitation wavelength in nm.
indicator (str) – Calcium indicator

I know this is an inherited problem from ImagingPlane but the indicator could be calcium, voltage, neurotransmitter, etc. right? So I think we should be a bit less specific to allow more flexibility. Ex.

• indicator (str) - Fluorescent indicator (eg. GCaMP7)

effector (str) - Opsin or channel membrane expressed by the targeted cell

Similarly with effector I think we should use more inclusive language like

• effector (str) - Light-activated effector protein expressed by the targeted cell (eg. ChR2)

I think it can be feasible to describe the shape of the stimulus on the single ROIs, as in the case of the spiral pattern, but if the shape of the stimulus varies in time for the single ROI or it's different for each ROI, I think it scales up to a level of complexity that it's not necessary in the NWB representation.

I disagree -- the precise spatiotemporal stimulus pattern could be a very important piece of data for any type of re-analysis of holographic optogenetic stimulation experiments. What if they stimulate the axon of a pre-synaptic neuron and then the dendrites of the post-synaptic neuron? Or vice versa? Importantly, as you pointed out, there are a variety of different stimulation patterns that one could use, and they grow complex as you scale the number of ROIs and sequential rounds of stim. I think it is reasonable to expect that as this technology matures, scientists and engineers will continue to explore different patterns of stimulation and flesh out the space of possibilities.

With that in mind, I think it might be more appropriate to model the HolographicStimulationSeries after pynwb.ophys.TwoPhotonSeries with a 4D data array. Then you could link to the HolographicStimulationTarget directly without needing the HolographicStimulusSite or HolographicStimulusPattern as intermediates.

[alessandratrapani]

Similarly with effector I think we should use more inclusive language like

• effector (str) - Light-activated effector protein expressed by the targeted cell (eg. ChR2)

Yes, thank you for the suggestion! "Opsin or channel membrane" was already general enough for me, but I surely have a bias on these things. I like your description more.

[alessandratrapani]

With that in mind, I think it might be more appropriate to model the HolographicStimulationSeries after pynwb.ophys.TwoPhotonSeries with a 4D data array. Then you could link to the HolographicStimulationTarget directly without needing the HolographicStimulusSite or HolographicStimulusPattern as intermediates.

I generally agree with you, the best way to describe the stimulus pattern is to recreate the hologram, i.e. a 4D image.
But I also wonder if, for the type of data we have right now - which is basically just a categorical description plus some parameters that describe the stimulus for each ROI - the reconstruction of the hologram would be a complex model and possibly prone to errors related to the type of model we use to reconstruct the hologram. This, of course, would not be the case if the data provided by the client were already the 4D image/hologram or if they could provide the model for the generation of the hologram (basically the SLM control).
This is why I thought a TimeSeries for the laser power deliver to the specific ROI and the link to the stimulus pattern description would be more feasible.

Nevertheless, since I don't have a strong position with this, I am open to any type of implementation, and maybe we can consult directly the point person for the holographic data in the Pinto lab and in the Adesnik lab. What do you think?

[pauladkisson]

I generally agree with you, the best way to describe the stimulus pattern is to recreate the hologram, i.e. a 4D image.
But I also wonder if, for the type of data we have right now - which is basically just a categorical description plus some parameters that describe the stimulus for each ROI - the reconstruction of the hologram would be a complex model and possibly prone to errors related to the type of model we use to reconstruct the hologram.

Agree. I think you've really exposed the core issue here, which is a tension between designing for the very simple stimulus patterns for which we have examples (spiral from Pinto and whole ROI from Adesnik) vs. designing for the much more complex stimulus patterns that might arise in the future.

Although I usually prefer to design with concrete examples in mind, I think we have an opportunity here to encourage good data practices by suggesting/requiring labs to provide the original stimulus hologram. It's important to remember that NWB isn't just a way to store data but also essentially a recommendation of what kinds of data/metadata are important to keep track of. This notion is really reinforced by perspective on neuroscience data standardization with NeurodataWithoutBorders Section 4.13. An indirect benefit of using NWB is improved data awareness.

That being said, of course it's a big problem if labs don't have their stimulus hologram recorded and no easy way of reconstructing it. So, let's ask the Adesnik lab tomorrow and Pinto lab (somehow) and see what they think about all this and what kind of data they actually have.

[alessandratrapani]

It's important to remember that NWB isn't just a way to store data but also essentially a recommendation of what kinds of data/metadata are important to keep track of. This notion is really reinforced by perspective on neuroscience data standardization with NeurodataWithoutBorders Section 4.13. An indirect benefit of using NWB is improved data awareness.

Very good point! I am quite sure that for Adesnik lab they would have the original stimulus hologram since the scope of the work is mainly to propose a new type of technology.

[alessandratrapani]

I am reporting here some consideration from Professor Pinto:

4D-hologram is more complete, but it comes with a few non-straightforward decisions:

a. The resolution:

You could do it in the native resolution of the spatial light modulator (SLM) generating the hologram, but it would then be tough to convert to imaging space. This is not just because resolutions are different, but also because they are going through different optical paths. Also, the effective pattern in the tissue will be different than the SLM because of the optics, and the best we can is to just estimate this in separate experiments. t is also not straightforward, since the patterns are scanned typically faster than single imaging frames.

b. Holographic patterns:

In our case, the holographic patterns are actually additionally scanned in a spiral, so they are never static. It would be hard to go to full holography space to which region of interest you are actually stimulating.

c. Stimulus parameter setting:

In the Bruker holography software the holographic pattern is designed by setting the parameters and defining x-y-z coordinates for each, so storing those instead of the hologram might in the end be easier to reproduce.

In their Datajoint pipeline they used a similar approach as we proposed in solution 2: Store a time series for the stimulus power delivered for each ROI plus a description of the type of stimulus.

We have a time series in imaging frame coordinates making it more straightforward for downstream analysis, and I think as long as parameters are thoroughly documented it should be good for reproducibility. The tricky part with this solution is making it generalizable for other types of holography, if that’s something desirable for you. For example, another popular way of doing it is using temporal focusing to generate pancake-shaped holography spots to fill the cell. They are not moved in spirals, but I'm not sure which additional parameters you need to define these.

In light of these considerations, I suggest staying within option 2 and additionally storing the SLM mask (if it is something we can actually request) along with the description. What do you all think?

I will iterate again on this once we have the opinion from Adesnik lab.

[pauladkisson]

a. The resolution:

You could do it in the native resolution of the spatial light modulator (SLM) generating the hologram, but it would then be tough to convert to imaging space. This is not just because resolutions are different, but also because they are going through different optical paths. Also, the effective pattern in the tissue will be different than the SLM because of the optics, and the best we can is to just estimate this in separate experiments. t is also not straightforward, since the patterns are scanned typically faster than single imaging frames.

This is the real problematic one imo. Although I am a little bit confused how they can easily access the stim pattern per ROI in the imaging space but can't access it in the larger FOV. It should be the same time series just localized to an ROI right? I guess what they recorded are the commanded stim patterns rather than the measured stim patterns? If so that should be fine, but it might be nice to clarify in the docs/names.

[alessandratrapani]

I made a few changes to simplify the structure: the effector is now a property of HolographicStimulusSite next to its excitation lambda (which makes more sense to me) and removed HolographicStimulusTarget consequently (we would have a PlaneSegmentation indexed by roi in the HolographicSeries); I also add few more proposal for the stimulus pattern representation.

HolographicStimulusPattern

to be discussed. Proposals:

1. 'summary image' that visually describes the excitation pattern of a single ROI
2. name, description and a DynamicTable with the descriptive parameters
3. functional description (equation representing the evolution in space of the stimulus)
4. 4D image - hologram
5. build an extension for each pattern

(personal opinion: I think that 2. and 5. are the more feasible options for the data we have now)

HolographicStimulusSite

Bases: OptogeneticStimulusSite
Holographic optogenetic stimulus site.
Parameters:

• name (str) – The name of this stimulus site.
• device (Device) – The device that was used.
• description (str) – Description of site.
• excitation_lambda (float) – Excitation wavelength in nm.
• effector (str) - Light-activated effector protein expressed by the targeted cell (eg. ChR2)
• location (str) – Location of the stimulation site.
• stimulus_pattern (HolographicStimulusPattern)
• (possibly add ImageSegmentation link)

HolographicSeries

Bases: OptogeneticSeries
Holographic optogenetic stimulus. The data field is in units of watts.
Parameters:

• name (str) – The name of this TimeSeries dataset
• data (ndarray or list or tuple or Dataset or Array or StrDataset or HDMFDataset or AbstractDataChunkIterator or DataIO or TimeSeries) – The data values. May be 1D or 2D. The first dimension must be time. The optional second dimension represents ROIs
• rois (DynamicTableRegion) – a table region corresponding to the ROIs that were stimulated (link that allows you to reference specific rows of PlaneSegmentation table by row indices.)
• site (HolographicStimulusSite) – The site to which this stimulus was applied.
• resolution (float) – The smallest meaningful difference (in specified unit) between values in data
• conversion (float) – Scalar to multiply each element in data to convert it to the specified unit
• timestamps (ndarray or list or tuple or Dataset or Array or StrDataset or HDMFDataset or AbstractDataChunkIterator or DataIO or TimeSeries) – Timestamps for samples stored in data
• starting_time (float) – The timestamp of the first sample
• rate (float) – Sampling rate in Hz
• comments (str) – Human-readable comments about this TimeSeries dataset
• description (str) – Description of this TimeSeries dataset
• control (Iterable) – Numerical labels that apply to each element in data
• control_description (Iterable) – Description of each control value
• offset (float) – Scalar to add to each element in the data scaled by ‘conversion’ to finish converting it to the specified unit.

Question:
If instead of having a time series for the stimulus representation we have time intervals (from the dataset provided by the lab), how would you proceed? Would you reconstruct the time series info from time intervals or would you prefer having a property to store the time intervals (keep in mind that both datasets we have so far use time intervals).

[pauladkisson]

HolographicStimulusPattern to be discussed. Proposals:

'summary image' that visually describes the excitation pattern of a single ROI
name, description and a DynamicTable with the descriptive parameters
functional description (equation representing the evolution in space of the stimulus)
4D image - hologram
build an extension for each pattern
(personal opinion: I think that 2. and 5. are the more feasible options for the data we have now)

Seems like we should try to make 2 work for both stimulus patterns, with 5 as a fallback.

[pauladkisson]

Question:
If instead of having a time series for the stimulus representation we have time intervals (from the dataset provided by the lab), how would you proceed? Would you reconstruct the time series info from time intervals or would you prefer having a property to store the time intervals (keep in mind that both datasets we have so far use time intervals).

I think we should keep the time series representation since it's more general. Although both labs that we're working with right now use fixed laser power per ROI, there is nothing technical preventing them from using continuously varying laser power (I think). Since it is easy to transform time intervals into a time series but hard to do with the reverse, I think we should stick with timeseries.

[alessandratrapani]

Thank a lot for the feedbacks @pauladkisson

[alessandratrapani]

While testing the extension, we realized we could not add a dynamic table or a generic NWBcontainer to the stimulus module.
We will extend the StimulusPattern from TimeSeries (that can be added to the stimulus module), with the possibility of leaving data empty and defining the specific attributes needed for the Spiral and the TemporalFocusing patterns (the two of them being different objects). Suppose an external user wants to add a different stimulus pattern. In that case, it can do so by using the data property and additional description or raising an issue to implement an extension.
Am I describing it correctly, @CodyCBakerPhD?

[CodyCBakerPhD]

@weiglszonja Also had a good idea about how to create a general group without overriding the NWBFile object; I'll let her describe the specifics, it could offer an alternative to putting things in the nwbfile.stimulus area

[weiglszonja]

Yeah, so the idea is to add a new container that extends LabMetaData, which is an NWBContainer under "general", so all the holographic related data could be stored in that container.

For reference this is how ndx-photometry does it https://github.com/catalystneuro/ndx-photometry/blob/7ea9d755ceac9524125f50ab528b403b135c4530/src/spec/create_extension_spec.py#L333

[alessandratrapani]

updated schema:

[alessandratrapani]

I would like to propose a change for the name of the extension. Instead of holographic-stimulation I think it's better to call it with a more generic denomination for patterned optogenetic stimulation:

1. ndx-patterned-ogen (I prefer this one)
2. ndx-patterned-phostostimulation
3. ndx-patterned-optogenetics

What do you think about it?

[weiglszonja]

I don't have a strong opinion about this, if you feel this extension should be more general we can go with ndx-patterned-ogen or ndx-patterned-optogenetics.

Would you also change the neurodata types as well? Or Holographic* would be a "sub" type of patterned stimulation?

[weiglszonja]

I also wanted to mention here https://github.com/bendichter/ndx-optogenetics , maybe for Device it can be relevant here?

[CodyCBakerPhD]

Haven't both labs referred to what do as holographic? Or are you saying the neurodata types would still have holographic references, just the overall name of the extension would be generalized?

Normally I'd suggest keeping such an extension within a targeted scope specific to the new data we're modeling

[CodyCBakerPhD]

I also wanted to mention here https://github.com/bendichter/ndx-optogenetics

That is yet another extension for ogen, largely for trying to satisfy some particular users criticism of the lack of metadata surrounding the core ogen data types, but even that extension has been said by some to fall short (something about 'titer' parameters if I recall)

and I would say if the Fiber Implant, etc aren't relevant to holographic specifically then they should be kept as their own extension just to keep the scope of our current needs manageable

[alessandratrapani]

Yes, I just came across the issue. I think that there are some good reasonings that we should keep in mind:

currently the stimulation wavelength is an attribute of the OptogeneticStimulusSite class which is used as an attribute of the OptogeneticSeries class --- the wavelength is not specific to a site, but to the actual stimulation series, so it should be moved to OptogeneticSeries.

Further, the naming for the wavelength is excitation_lambda, “excitation” is a hypothesized result and not a parameter of the stimulation, further, the nomenclature isn't really consistent with optogeneticcs where opsins are commonly “activated” or “inactivated”, which in turn can lead to either neuronal “excitation” or “inhibition” depending on the opsin. So the name itself should also be changed to something like e.g. wavelength to avoid confusion.

OptogeneticStimulusSite is currently listed as having an attribute named Device. This is a bit backwardds since the stimulation happens with the involvement of one or multiple devices, which then have locations. Things necessitate places, places don't necessitate things. So OptogeneticSeries.Device.OptogeneticStimulusSite instead of the current OptogeneticSeries.OptogeneticStimulusSite.Device would be the correct way to go about this in principle --- though for the specific meaning of Device OptogeneticSeries.Device without any relationship to the OptogeneticStimulusSite would be the way to go

Device seems to refer specifically to the device which produces the stimulation, so the Laser/LED. This device doesn't have any experimentally meaningful location. It's somewhere in the lab.

The other observations are specific to optogenetic with fiber implants, but I also agree that you don't really need OptogeneticStimulusSite since location could be a property of the series directly as well as pattern in our case.

That said I think it's better to keep it separate from our extension because that type of photostimulation is done with fiber implants, ours with 2p microscope (the same way fiber photometry is different from two photon imaging).

All of these consideration might be good for a proposal for modifying the core ogen, right?

[CodyCBakerPhD]

All of these consideration might be good for a proposal for modifying the core ogen, right?

Yes, over a longer time scale once each module has stabilized in its own right

[alessandratrapani]

I would like to discuss a new proposal for the extension that would include all the info we need for the holographic series and possibly tackle some of the issues related to the current optogenetic series.
Such as:

• the device being defined as a property of the stimulus site
• the lack of the effector information, as well as the type of photostimulation, both properties of the stimulus series itself
• renaming the excitation_lambda as stimulation_wavelength for clarity

Major changes:

• renaming the extension ndx-patterned-ogen (to be more inclusive of all optogenetic stimulus that is not holographic stimulation but could use specific beam pattern to stimulate specific ROIs).
• renaming the series PatternedOptogeneticSeries
• remove HolographicStimulusSite link
• renaming the pattern OptogeneticStimulusPattern
• include attributes to the series:
  a. the information related to the opsin used to exert the stimulation,
  b. the stimulation wavelength
  c. the stimulation type (excitatory/inhibitory)
• include a link for the Device directly in the series
• include a link for the OptogeneticStimulusPattern directly in the series

I know we are moving a bit far from the current implementation of the ogen module, which might create confusion. But I will leave this proposal here anyway for future discussion.

[pauladkisson]

renaming the extension ndx-patterned-ogen (to be more inclusive of all optogenetic stimulus that is not holographic stimulation but could use specific beam pattern to stimulate specific ROIs).

What exactly are you referring to here? Is there a paper that uses a stimulation pattern like this?

renaming the excitation_lambda as stimulation_wavelength for clarity

Makes sense to me. Remember, we also discussed excitation lambda being miss placed and miss named for imaging in ndx-microscopy. It might be reasonable to include optogenetics in that NEP as well.

include attributes to the series:
a. the information related to the opsin used to exert the stimulation,
b. the stimulation wavelength
c. the stimulation type (excitatory/inhibitory)

As pointed out in this issue, some metadata like the excitation lambda has been miss placed into the stimulus site instead of the stimulus series. As we discussed in ndx-microscopy, this problem also occurs with calcium imaging where meta-data is haphazardly thrown into the imaging plane object. However, I don't think that combining the stimulus site and stimulus series into one object is the appropriate solution. Instead, we should proceed as we did in ndx-microscopy: the stimulus site / imaging plane should be limited to only describe properties native to the actual site ex. location, effector/indicator, etc. Then, the other metadata can be moved either to the series object directly or to a separate object(s) that is referenced by the series object (ex. Microscope/Optic Channel in ndx-microscopy).

Specifically, I recommend that the holographic stimulus site object has a location, effector, and rois. The holographic series object would have a device, stimulus pattern, site, and excitation wavelength.

I also don't think we really need to specify the stimulation type, since effectors can have more diverse actions then just excitation or inhibition -- see this paper on using light-activated effector proteins for triggering intracellular signaling. I think it's better to just specify the effector by name, and then people can go look up what it does.

[alessandratrapani]

What exactly are you referring to here? Is there a paper that uses a stimulation pattern like this?

Nice review paper here gave me the idea of calling it ndx-patterned-ogen

Specifically, I recommend that the holographic stimulus site object has a location, effector, and ROIs.

I like this solution, and I also thought about it, but since the series dimensions are n_times x n_rois I thought it would be best to have rois directly in the series object. However, if you think it is not a problem, I am happy to change in favor of this solution.

The holographic series object would have a device, stimulus pattern, site, and excitation wavelength.

Except for changing the name from excitation wavelength to stimulation wavelength, right?

I also don't think we really need to specify the stimulation type, since effectors can have more diverse actions than just excitation or inhibition -- see this paper on using light-activated effector proteins for triggering intracellular signaling. I think it's better to just specify the effector by name, and then people can go look up what it does.

Thanks for the suggestion. I wasn't quite sure whether to include this or not, because I think it's an intrinsic feature of the effector. I will remove it.

[pauladkisson]

Thank you for the paper recommendation it was very informative, especially the summary table at the end. I agree that it is reasonable to incorporate non-holographic patterning methods into this extension such as 2 photon galvanometric mirror and auto acoustic deflectors –patterned optogenetics seems like a good name to me.

I like this solution, and I also thought about it, but since the series dimensions are n_times x n_rois I thought it would be best to have rois directly in the series object. However, if you think it is not a problem, I am happy to change in favor of this solution.

We do need to be careful about this one. What if one wanted to deliver stimulation to a subset of the total number of our rOIs for a given holographic series? Would you need to add zero for all of the ROIS that you're not stimulating in this particular series?

Except for changing the name from excitation wavelength to stimulation wavelength, right?

Yeah, just a typo on my part.

[alessandratrapani]

We do need to be careful about this one. What if one wanted to deliver stimulation to a subset of the total number of our rOIs for a given holographic series? Would you need to add zero for all of the ROIS that you're not stimulating in this particular series?

Yes, exactly. Or you can always create an roi table that indexes only the stimulated rois.