[Feature]: Improve embedding of namespaces on nwbExport #607
Assignees
Labels
category: enhancement
improvements of code or code behavior
priority: medium
non-critical problem and/or affecting only a small set of NWB users
status: need verification
potentially solved, but needs verification
topic: nwb-file
relates to how an exported nwb file is structured
Comments
ehennestad
added
category: enhancement
ehennestad
added a commit
that referenced
this issue
Nov 4, 2024
8 tasks
ehennestad
added
status: need verification
bendichter
added a commit
that referenced
this issue
Feb 11, 2025
…B file on export (#615) * Only include namespaces for types that are included in NWB file on export (Issue #607) * Add functionality for installing extensions * Minor fixes * Update comment * Add comment + print message when extension has been installed * Update installExtension.m * Fix changed variable name * Update matnwb_createNwbInstallExtension.m Update docstring * Create listNwbTypeHierarchy.m Add utility function for listing the type hierarchy of an nwb type * Add private method for embedding specifications to file on export * Fix variable name * Add workflow for updating nwbInstallExtension * Add option to save extension in custom location * Create InstallExtensionTest.m * Update docstring * Change dispExtensionInfo to return info instead of displaying + add test * Reorganize code into separate functions and add tests * Minor changes to improve test coverage * add nwbInstallExtension to docs * Update update_extension_list.yml Add schedule event for workflow to update nwbInstallExtension * Update downloadExtensionRepository.m Remove local function * Update docstring for nwbInstallExtension * Fix docstring indentation in nwbInstallExtension * Add doc pages describing how to use (ndx) extensions * Fix typo * Update +tests/+unit/InstallExtensionTest.m Co-authored-by: Ben Dichter <ben.dichter@gmail.com> * Update docs/source/pages/getting_started/using_extensions/generating_extension_api.rst Co-authored-by: Ben Dichter <ben.dichter@gmail.com> * Add docstrings for functions to retrieve and list extension info * Fix docstring formatting/whitespace * Update listExtensions.m Add example to docstring * Move static test methods into io.internal.h5 namespace Introduce some functions that will be useful later * Update writeEmbeddedSpecifications.m Add arguments block, fix function name * Add validateEmbeddedSpecifications * Update NwbFile.m Redefine listNwbTypes method, add validation of embedded namespaces * Create listEmbeddedSpecNamespaces.m * Update nwbExportTest.m * Update test for spec/namespace embedding * Update read_indexed_column.m * Add disclaimer in deleteGroup function * Update read_indexed_column.m * Fix broken test * add test-requirement * Fix: Ensure object is group before deleting * Fix error id * Add unittests for functions in io.internal.h5 namespace * Update nwbExportTest.m Added comments and a better test to test for warning with ID 'NWB:validators:MissingEmbeddedNamespace' * Fix failing tests --------- Co-authored-by: Ben Dichter <ben.dichter@gmail.com>
ehennestad
added a commit
that referenced
this issue
Feb 25, 2025
…ormance (#658) * Coarse fix for #595 * Only include namespaces for types that are included in NWB file on export (Issue #607) * Add functionality for installing extensions * Minor fixes * Update comment * Add comment + print message when extension has been installed * Update installExtension.m * Fix changed variable name * Update matnwb_createNwbInstallExtension.m Update docstring * Create listNwbTypeHierarchy.m Add utility function for listing the type hierarchy of an nwb type * Add private method for embedding specifications to file on export * Fix variable name * Update testCreateParsedType.m * Suppress output of nwbfile * Add workflow for updating nwbInstallExtension * Add option to save extension in custom location * Create InstallExtensionTest.m * Add .env file to replace env.mat * Update .gitignore * Update docstring * Change dispExtensionInfo to return info instead of displaying + add test * Reorganize code into separate functions and add tests * Minor changes to improve test coverage * add nwbInstallExtension to docs * Update update_extension_list.yml Add schedule event for workflow to update nwbInstallExtension * Update downloadExtensionRepository.m Remove local function * Update docstring for nwbInstallExtension * Fix docstring indentation in nwbInstallExtension * Add doc pages describing how to use (ndx) extensions * Fix typo * Update +tests/+unit/InstallExtensionTest.m Co-authored-by: Ben Dichter <ben.dichter@gmail.com> * Update docs/source/pages/getting_started/using_extensions/generating_extension_api.rst Co-authored-by: Ben Dichter <ben.dichter@gmail.com> * Add docstrings for functions to retrieve and list extension info * Fix docstring formatting/whitespace * Update listExtensions.m Add example to docstring * Move static test methods into io.internal.h5 namespace Introduce some functions that will be useful later * Update writeEmbeddedSpecifications.m Add arguments block, fix function name * Add validateEmbeddedSpecifications * Update NwbFile.m Redefine listNwbTypes method, add validation of embedded namespaces * Create listEmbeddedSpecNamespaces.m * Update nwbExportTest.m * Update test for spec/namespace embedding * Update read_indexed_column.m * Add disclaimer in deleteGroup function * Update read_indexed_column.m * Fix broken test * add test-requirement * Fix: Ensure object is group before deleting * Fix error id * Add unittests for functions in io.internal.h5 namespace * Suppress output * Remove duplicated code * Update fillClass.m Set Required "tag" for properties derived from attributes and links * Update fillClass/fillExport to add check for dependent & required properties on export * Add warnIfRequiredDependencyMissing If a required attribute depends on a non-required Dataset or Group, this method issues a warning if the dependent attribute (property) is missing when it's parent is present * Update generated types * Add default property values to generated type classes * Update warnIfPropertyAttributeNotExported to compare value with defult value before issuing warning * Add value for required property in VectorData * Add unit tests * Supress warning in test * Update tests.util.getPythonPath and remove similar code from +tests/+system/PyNWBIOTest.m * Update GenerationTest.m Add note * Suppress warnings in DataPipeTest use verifyWarning and verifyError to make code clearer Make testFilterOverride for better test granularity * Suppress output * Improve performance of if block Make check targeted to classes that have readonly prop * Remove try/catch * Use verifyWarning instead of custom warning manipulation * Updated neurodata type classes * Add unit test * Add shared fixture * Chnage name part 1/2 * Change name part2/2 * Suppress output * Add shared test fixture to tutorialtest * Update nwbtest.m * Update nwbtest.m - Use method for sorting suite by fixture - nwbClearGenerated + cleanup should be handled by test fixtures - Add option to skip generatimg coverage report * Refactor unit tests for test-schemas for better performance (#659) * Add TypesOutputFolder as property on NwbTypeGeneratorFixture class * Add abstract class for unit tests that tests test-schemas * Move test schemas to separate folder * Move test-schema Tests to tests.unit.schema namespace and subclass from tests.unit.abstract.SchemaTest * Update class names to match filenames * Replace getRandInd with builtin randi function * Reduce number of iterations in RegionViewTest * Restore file wrongly added to PR * Add Extension fixture * Refactor - slide suite sorting * Refactor misc unittests in "tests.unit" namespace (#660) * Change misc unittests from function- to class based * Change testCreateParsedType from function to class-based test suite * Fix generate types in folder set up by Shared fixture * Update dataStubTest.m * Move tutorial tests from tests.unit to tests.system.tutorial (#661) * Update nwbtest.m * Fix bug with clearing and regenerating types * Ignore generated types from test coverage * Rename ExtensionFixture to ExtensionGenerationFixture * Add docstring to ExtensionGenerationFixture * Rename ResetGeneratedTypesFixture to NwbClearGeneratedFixture * Rename NwbTypeGeneratorFixture to GenerateCoreFixture * Fix - add default input to NwbClearGeneratedFixture * Update nwbtest - Save generated reports to fixed directory * Add docs/reports to gitignore * Add .coverageignore file and local function for collecting list of files to produce code coverage for * Fix workflow for running tests (report output directory) * Update nwbExportTest.m Added comments and a better test to test for warning with ID 'NWB:validators:MissingEmbeddedNamespace' * Update branch name references from master to main * Change branch name from master to main in md and workflow files * Fix typo * Fix failing tests * Suppress more expected warnings and outputs * Add test * Update test to test different data types * Make local function getEmptyRepresentation more explicit * Remove unused code from test * Use shared fixture in WarningsTest * Update dataStubTest, include new test in test methods block * Add shared fixture to new test MustBeH5FileTest * Add abstract NwbTestCase class with utility methods that can be reused * Suppress output from various tests * Fix failing test * Remove commented code block from ObjectView class * Add utility method to NwbTestCase class * Fix bug and add test * Add "version" as property to Namespace class * Add routine for writing a function for retrieving version of generated namespace * Regenerate core with version numbers * Update generateRstForNeurodataTypeClasses.m Ignore version function when creating list of neurodata types to document * Add version specificity to cache containing map of required properties * Delete GenerateSpecificationMixin.m * Split test class setup into methods with single responsibilities * Add factory function to create NwbFile for unit testing * Clean up nwbExport test * Clean up a few more test classes * Add fixture for setting environment variables used in test suite and remove env.mat * Update PyNWBIOTest.m Add shared fixture * Fix wrong shape of data in PhotonSeriesIOTest * Update PyNWBIOTest.py --------- Co-authored-by: Ben Dichter <ben.dichter@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What would you like to see added to MatNWB?
All extension namespaces in the cache (namespaces folder), will get embedded in the specs of an NWB file when running nwbExport. This can lead to files containing embedded specifications that are not relevant to any of the data types in the file.
Is your feature request related to a problem?
One recorded problem of this in #603
What solution would you like?
Implement a system detecting what namespaces a user actually used and embedding schemas off of that.
Do you have any interest in helping implement the feature?
Yes.
Code of Conduct
The text was updated successfully, but these errors were encountered: