Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load data from databroker and create an in-memory H5 #17

Open
leofang opened this issue Jan 15, 2019 · 3 comments
Open

load data from databroker and create an in-memory H5 #17

leofang opened this issue Jan 15, 2019 · 3 comments

Comments

@leofang
Copy link
Contributor

leofang commented Jan 15, 2019

Starting from v2.9.0, h5py supports in-memory H5, which means one can allocate a segment of shared memory, within which a H5 object can live. If the frontend can do this, it will help avoid reading large data from disk in the backend.

Thoughts:

  • to actually save time, we want to reuse the same H5 many times before closing the shared memory. Is this the most common use case?
  • before killing the in-memory object, should we write it to disk? (I think so)
@leofang
Copy link
Contributor Author

leofang commented May 14, 2019

Caution: this will go wrong if the GUI uses multiple nodes...

@leofang
Copy link
Contributor Author

leofang commented Feb 1, 2020

@mrakitin this is what I mentioned the other day

@mrakitin
Copy link

mrakitin commented Feb 1, 2020

For the first point, I think it makes sense to reuse.
+1 for writing to disk in case the dataset should be available later.

I'd ask @tacaswell for his thoughts as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants