Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HashStore 1.1.0 Release #84

Merged
merged 456 commits into from
Oct 2, 2024
Merged
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
456 commits
Select commit Hold shift + click to select a range
63ad53c
Fix bug in 'hashstoreclient' where formatid is always 'None' if no fo…
doulikecookiedough May 23, 2024
2ed6340
Merge pull request #95 from DataONEorg/feature-94-client-bugfix
doulikecookiedough May 23, 2024
fcec590
Add addition logging statements to 'tag_object', 'find_object' and '_…
doulikecookiedough May 23, 2024
a520a5f
Fix inaccurate logging msg in 'delete_object' and typos
doulikecookiedough May 24, 2024
3368bec
Further revise and add logging statements to 'store_object' and 'tag_…
doulikecookiedough May 24, 2024
277370e
Merge pull request #96 from DataONEorg/feature-93-more-logging
doulikecookiedough May 24, 2024
2db9a8c
Revise debug statement for 'tag_object'
doulikecookiedough May 24, 2024
af4e7b7
Rename variables in 'tag_object' and update debugging message
doulikecookiedough May 24, 2024
fffa0e5
Revise '_verify_hashstore_references' method and update pytests
doulikecookiedough May 24, 2024
6521b10
Test: add while block to 'store_object' to wait before calling 'tag_o…
doulikecookiedough May 24, 2024
a68a2fa
Revert 'tag_object' changes
doulikecookiedough May 27, 2024
13a4b45
Add new pytest in 'filehashstore_interface' test module to check pyth…
doulikecookiedough May 27, 2024
ba76b51
Test: Implement check to see if multiprocessing is being used to sele…
doulikecookiedough May 27, 2024
0259972
Test: Revise 'tag_object' synchronization by adding while statement f…
doulikecookiedough May 27, 2024
9a14af5
Test: Set 'USE_MULTIPROCESSING' global variable in HashStoreClient
doulikecookiedough May 27, 2024
57b37bc
Remove commented out code, retaining test changes for 'tag_object' to…
doulikecookiedough May 27, 2024
c55dae1
Add todo items for implementing multiprocessing locks
doulikecookiedough May 27, 2024
c054fe7
Test: Refactor 'tag_object' to check existence for files within if-el…
doulikecookiedough May 27, 2024
ecf0517
Improve clarity in 'tag_object' else block by throwing new custom exc…
doulikecookiedough May 28, 2024
1de704b
Reduce time spent in 'tag_object' try-finally block by moving code to…
doulikecookiedough May 28, 2024
93f00fe
Move preparation of refs file back into try statement in 'tag_object'
doulikecookiedough May 29, 2024
a68da20
Revise 'tag_object' to synchronize with threading.Condition
doulikecookiedough May 30, 2024
36b140c
Revise 'tag_object' to also synchronize with multiprocessing.Condition
doulikecookiedough May 30, 2024
9ddc79d
Fix bug with 'tag_object' multiprocessing sync adding 'cid' to the in…
doulikecookiedough May 30, 2024
d8df582
Rename custom exception from 'PidRefsExistsError' to 'PidAlreadyExist…
doulikecookiedough May 30, 2024
b3da712
Revise comment and add note in 'test_filehashstore_interface' module
doulikecookiedough May 30, 2024
fccc792
Remove unused import
doulikecookiedough May 30, 2024
ad55464
Merge pull request #100 from DataONEorg/bug-97-tagobject-cidrefs
doulikecookiedough May 30, 2024
78ddad7
Add new pytest (failing) to confirm issue of tmp files remaining
doulikecookiedough May 30, 2024
5ac31ce
Update docstrings in 'test_filehashstore_interface' for new pytests
doulikecookiedough May 30, 2024
cf21ee2
Revise 'tag_object' to create tmp refs files when required
doulikecookiedough May 30, 2024
bf904c1
Merge pull request #102 from DataONEorg/bug-101-tmp-refsfiles
doulikecookiedough May 30, 2024
5aab681
Remove organizing lock TODO item, add new threading 'Condition' for o…
doulikecookiedough May 31, 2024
bf56981
Refactor 'store_object' to use condition synchronization
doulikecookiedough May 31, 2024
9c483f2
Implement 'multiprocessing' lock for 'store_object'
doulikecookiedough May 31, 2024
c340268
Refactor 'store_metadata' to use condition synchronization
doulikecookiedough May 31, 2024
763b601
Implement 'multiprocessing' lock for 'store_metadata'
doulikecookiedough May 31, 2024
73fc7a9
Refactor 'delete_object' to use condition synchronization
doulikecookiedough May 31, 2024
d6f19f2
Implement 'multiprocessing' lock for 'delete_object'
doulikecookiedough May 31, 2024
d8c6117
Implement thread and multiprocessing synchronization for 'delete_meta…
doulikecookiedough May 31, 2024
d368d04
Clean up 'store_object' method
doulikecookiedough May 31, 2024
8015262
Clean up 'tag_object' method
doulikecookiedough May 31, 2024
1d0a076
Clean up 'store_metadata' method
doulikecookiedough May 31, 2024
5742c78
Clean up 'delete_object' method
doulikecookiedough May 31, 2024
02d2ef7
Clean up 'delete_metadata' method
doulikecookiedough May 31, 2024
2faeefb
Revise 'hashstoreclient' module to set global env 'USE_MULTIPROCESSIN…
doulikecookiedough May 31, 2024
f1711fc
Refactor 'filehashstore' to check for global env 'USE_MULTIPROCESSING…
doulikecookiedough May 31, 2024
c9ee2ea
Remove unused 'time' module and related code
doulikecookiedough May 31, 2024
92cbb2c
Update 'README.md' with instructions on how to switch HashStore from …
doulikecookiedough Jun 3, 2024
f1fd49d
Revise debug statements in 'tag_object' when a cid refs file is updated
doulikecookiedough Jun 3, 2024
1874fb1
Merge pull request #103 from DataONEorg/feature-98-multiprocessing-locks
doulikecookiedough Jun 3, 2024
5970ed8
Update 'pyproject.toml' author order
doulikecookiedough Jun 4, 2024
95e78ae
Add skeleton for 'setup' tools and a test module before swapping it w…
doulikecookiedough Jun 4, 2024
847863b
Update 'pyproject.toml' with new section 'tool.poetry.scripts' and ad…
doulikecookiedough Jun 4, 2024
9e20843
Delete WIP files (setup.py, command_line.py) and revise 'hashstorecli…
doulikecookiedough Jun 4, 2024
6b2f718
Update 'README.md' section for how to use the hashstore client
doulikecookiedough Jun 4, 2024
f28629f
Merge pull request #104 from DataONEorg/feature-92-client-syntax
doulikecookiedough Jun 4, 2024
099b7bf
Refactor 'store_metadata' to synchronize based on doc name (hash of t…
doulikecookiedough Jun 6, 2024
b5e1dc8
Refactor '_put_metadata' to receive metadata doc name and update pytest
doulikecookiedough Jun 6, 2024
baf83ea
Refactor 'delete_metadata' to sync based on doc name
doulikecookiedough Jun 6, 2024
6a9cc2b
Refactor 'delete_object' to call 'delete_metadata' to remove metadata…
doulikecookiedough Jun 6, 2024
987bc69
Rename 'metadata_locked_pids' to 'metadata_locked_docs' to improve cl…
doulikecookiedough Jun 6, 2024
597a1ff
Fix bug in 'delete_metadata' where sync value was not updated to 'pid…
doulikecookiedough Jun 6, 2024
229a11a
Fix linting errors related to unused variables and typos
doulikecookiedough Jun 6, 2024
a444a31
Merge pull request #105 from DataONEorg/feature-99-storedelmeta-update
doulikecookiedough Jun 7, 2024
aadaefc
Update 'HashStoreClient' class to use different module names based on…
doulikecookiedough Jun 11, 2024
76d4365
Merge pull request #107 from DataONEorg/feature-106-pgdbcode-cleanup
doulikecookiedough Jun 11, 2024
bc31278
Merge branch 'main' into develop
doulikecookiedough Jun 14, 2024
c761585
Update 'hashstore' interface for 'verify_object' - no longer returns …
doulikecookiedough Jun 17, 2024
eb73b51
Refactor '_verify_object_information' to throw new custom exception '…
doulikecookiedough Jun 17, 2024
b75f05e
Delete redundant code from 'store_object', refactor '_move_and_get_ch…
doulikecookiedough Jun 17, 2024
c2878ca
Removed 'PidObjectMetadataError' import from 'test_filehashstore_inte…
doulikecookiedough Jun 17, 2024
4f3e198
Revise '_clean_algorithm' to throw new custom exception 'UnsupportedA…
doulikecookiedough Jun 17, 2024
9d405e8
Implement to-do item in 'verify_object' when called with an algo to v…
doulikecookiedough Jun 17, 2024
6697404
Fix minor bug in 'hashstoreclient' when checking for knbvm testflag
doulikecookiedough Jun 17, 2024
2cf9b45
Refactor 'verify_hashstore_references' to throw custom exception clas…
doulikecookiedough Jun 17, 2024
97cd1e4
Create new 'filehashstore_exceptions' module and refactor filehashsto…
doulikecookiedough Jun 17, 2024
1adadc5
Reorganize 'filehashstore_exceptions' module and add missing class mo…
doulikecookiedough Jun 17, 2024
777be91
Cleanup 'test_filehashstore_interface' module and add pytest
doulikecookiedough Jun 17, 2024
d63aea8
Merge pull request #110 from DataONEorg/feature-109-verifyobj-update
doulikecookiedough Jun 17, 2024
18cb8bf
Update README.md and filehashstore class docstring for accuracy
doulikecookiedough Jun 24, 2024
95e7baf
Update store directory used in tests to end with 'hashstore' to reduc…
doulikecookiedough Jun 25, 2024
f0274e4
Revise 'FileHashStore' init process to check for specific hashstore r…
doulikecookiedough Jun 25, 2024
47996d4
Add new pytests to check init with conflicting and non-conflicting di…
doulikecookiedough Jun 25, 2024
8d120e0
Update old placeholder sysmeta namespace value in tests and examples …
doulikecookiedough Jun 25, 2024
589d077
Merge pull request #115 from DataONEorg/feature-112-hashstore-initupdate
doulikecookiedough Jun 25, 2024
3063944
Update 'hashstore' interface for 'find_object'
doulikecookiedough Jun 25, 2024
c3faaaa
Refactor 'find_object' to return an object_info_dict containing the n…
doulikecookiedough Jun 25, 2024
4dfb4f5
Update 'hashstoreclient's 'find_object' output and update pytest
doulikecookiedough Jun 25, 2024
bb292ef
Fix bug in 'hashstoreclient' where object_path was not referenced cor…
doulikecookiedough Jun 25, 2024
581bd8e
Add new client pytest for 'find_object' when sysmeta exists
doulikecookiedough Jun 25, 2024
a032caf
Update new 'hashstoreclient' test docstrings
doulikecookiedough Jun 25, 2024
627a63b
Merge pull request #116 from DataONEorg/feature-113-findobj-update
doulikecookiedough Jun 25, 2024
1a66707
Revise 'tag_object' to throw new custom exception 'HashStoreRefsAlrea…
doulikecookiedough Jun 25, 2024
dc5973a
Refactor pytest 'test_store_object_duplicates_threads's 'store_object…
doulikecookiedough Jun 25, 2024
04f7379
Cleanup 'test_filehashstore_interface' module
doulikecookiedough Jun 25, 2024
373a870
Merge pull request #117 from DataONEorg/feature-114-existing-hashstor…
doulikecookiedough Jun 25, 2024
8b1b24a
Update 'pyproject.toml' to replace specific versions (^) with minimum…
doulikecookiedough Jul 1, 2024
1f87869
Update 'hashstore's '__init__.py' docstring for clarity
doulikecookiedough Jul 1, 2024
9e53c5b
Update 'README.md' author section and hashstore creation example
doulikecookiedough Jul 1, 2024
eeecc1c
Update 'README.md' and 'HashStoreFactory's 'get_hashstore' method for…
doulikecookiedough Jul 1, 2024
17e2eff
Update 'CONTRIBUTING.md' to be more friendly by including emojis
doulikecookiedough Jul 1, 2024
e45d038
Fix incorrect module_name in README.md example for instantiating a Ha…
doulikecookiedough Jul 2, 2024
2ad42aa
Move 'threading' and 'multiprocessing' variable declaration to '__ini…
doulikecookiedough Jul 2, 2024
83d2421
Update 'hashstoreclient' to set multiprocessing flag based on the te…
doulikecookiedough Jul 2, 2024
8b1e194
Merge pull request #120 from DataONEorg/bug-119-hashstore-import
doulikecookiedough Jul 2, 2024
308fd32
Merge branch 'develop' into feature-118-citation-package
doulikecookiedough Jul 2, 2024
8d9d654
Temporarily revert version back to '1.0.0'
doulikecookiedough Sep 8, 2024
94f8d13
Merge pull request #127 from DataONEorg/feature-118-citation-package
doulikecookiedough Sep 8, 2024
6d754bc
Refactor '_check_string' method to use inspect library to grab callin…
doulikecookiedough Sep 8, 2024
d81ebc7
Fix functions calling '_check_string'
doulikecookiedough Sep 8, 2024
10b5de5
Add new pytest for '_check_string' and also revise '_check_string' to…
doulikecookiedough Sep 8, 2024
e345834
Revise doc strings
doulikecookiedough Sep 8, 2024
37a38d8
Fix inaccurate docstrings in '_store_and_validate_data' and related m…
doulikecookiedough Sep 8, 2024
8129050
Revise docstrings to resolve linting warnings
doulikecookiedough Sep 8, 2024
5443b39
Merge pull request #128 from DataONEorg/feature-121-safeguard-pid
doulikecookiedough Sep 8, 2024
9a9057b
Refactor 'ObjectMetadata' class from namedtuple to be a dataclass
doulikecookiedough Sep 9, 2024
7f44395
Extract 'ObjectMetadata' class from 'HashStore' to 'FileHashStore', r…
doulikecookiedough Sep 9, 2024
fb31bb5
Revise docstrings in hashstore interface for accuracy
doulikecookiedough Sep 9, 2024
8411d5a
Remove unused import
doulikecookiedough Sep 9, 2024
6193329
Cleanup 'test_filehashstore' for unused variable
doulikecookiedough Sep 9, 2024
00c9dcd
Merge pull request #129 from DataONEorg/feature-126-objectmetadata-da…
doulikecookiedough Sep 9, 2024
78e8af7
Refactor '_validate_properties' in filehashstore to return a properti…
doulikecookiedough Sep 10, 2024
641cf7d
Merge pull request #130 from DataONEorg/feature-125-improve-init
doulikecookiedough Sep 10, 2024
2869c0c
Remove 'find_object' from the hashstore interface and re-organize pyt…
doulikecookiedough Sep 10, 2024
b54db03
Cleanup unused code and docstrings
doulikecookiedough Sep 10, 2024
249f9f8
Move placement of 'find_object' function in 'filehashstore'
doulikecookiedough Sep 10, 2024
ebb96f6
Remove 'find_object' from hashstore client and respective pytests
doulikecookiedough Sep 10, 2024
9138892
Update 'README.md'
doulikecookiedough Sep 10, 2024
74d446f
Merge pull request #131 from DataONEorg/feature-123-remove-findobject
doulikecookiedough Sep 10, 2024
32b9db8
Fix formatting inconsistencies caused due to pycharm settings
doulikecookiedough Sep 10, 2024
7cda167
Update 'delete_object' in hashstore interface to only accept a pid
doulikecookiedough Sep 10, 2024
2d674a4
Refactor 'delete_object' by extracting method 'delete_object_only' an…
doulikecookiedough Sep 10, 2024
1f5868d
Add docstring to 'delete_object_only' method and re-organize pytests
doulikecookiedough Sep 10, 2024
dff873d
Re-organize 'delete_object_only' method in filehashstore
doulikecookiedough Sep 10, 2024
eaabe3a
Extract '_get_hashstore_pid_refs_path' from '_resolve_path' and updat…
doulikecookiedough Sep 11, 2024
d42d916
Extract '_get_hashstore_cid_refs_path' from '_resolve_path' and updat…
doulikecookiedough Sep 11, 2024
d94b39c
Extract '_get_hashstore_data_object_path' from '_resolve_path'
doulikecookiedough Sep 11, 2024
232cd4d
Extract '_get_hashstore_metadata_path' from '_resolve_path'
doulikecookiedough Sep 11, 2024
7ed22b4
Delete '_resolve_path' method and refactor affected code and pytests
doulikecookiedough Sep 11, 2024
aabb690
Rename '_build_path' to '_build_hashstore_data_object_path' and updat…
doulikecookiedough Sep 11, 2024
2321b6f
Refactor '_build_hashstore_data_object_path' by removing redundant si…
doulikecookiedough Sep 11, 2024
5446ee3
Clean up '_get_hashstore_data_object_path' and '_get_hashstore_metada…
doulikecookiedough Sep 11, 2024
4218560
Add context and revise '_shard' docstring so that it is easier to und…
doulikecookiedough Sep 11, 2024
5215cc4
Fix docstring length
doulikecookiedough Sep 11, 2024
d1644d0
Rename 'hashstore' public APi method 'verify_object' to 'delete_inval…
doulikecookiedough Sep 11, 2024
7535949
Make 'delete_object_only' a private method by renaming it to '_delete…
doulikecookiedough Sep 11, 2024
ca3f1b8
Make 'find_object' a private method by renaming it to '_find_object' …
doulikecookiedough Sep 11, 2024
93d81eb
Revise docstrings to resolve linting warnings
doulikecookiedough Sep 11, 2024
79ab4f0
Cleanup pytests for unused and redundant code
doulikecookiedough Sep 12, 2024
791c17e
Clean up methods '_store_data_only', '_find_object' and '_store_and_v…
doulikecookiedough Sep 12, 2024
15e3949
Merge pull request #132 from DataONEorg/feature-124-verifyobject-update
doulikecookiedough Sep 12, 2024
1c7b988
Add new custom exception 'StoreObjectForPidAlreadyInProgress', refact…
doulikecookiedough Sep 12, 2024
4a18ea8
Refactor 'tag_object' scenario when both refs files exist to also inc…
doulikecookiedough Sep 12, 2024
aba0b74
Refactor 'tag_object' scenario when pid refs exists to throw exceptio…
doulikecookiedough Sep 12, 2024
ddc1793
Remove return value from 'tag_object' in hashstore interface, refacto…
doulikecookiedough Sep 12, 2024
4bc31cf
Rename synchronization variables for accuracy
doulikecookiedough Sep 12, 2024
69ed1d2
Refactor 'tag_object' by extracting method '_synchronize_referenced_l…
doulikecookiedough Sep 12, 2024
414657e
Add new synchronization variables for pids to ensure thread safety fo…
doulikecookiedough Sep 13, 2024
66a0035
Rename inaccurate sync variables, add missing docstring params and ad…
doulikecookiedough Sep 13, 2024
7239600
Add pid synchronization to 'tag_object' and rename '_synchronize_refe…
doulikecookiedough Sep 13, 2024
50838cc
Refactor 'tag_object' by extracting new method '_release_object_locke…
doulikecookiedough Sep 13, 2024
5d17f34
Add debug logging to release sync methods and refactor affected metho…
doulikecookiedough Sep 14, 2024
c4124da
Rename 'delete_invalid_object' to 'delete_if_invalid_object' and upda…
doulikecookiedough Sep 14, 2024
e37aa17
Refactor 'tag_object' by moving sync methods into try block
doulikecookiedough Sep 14, 2024
a769130
Refactor 'tag_object' by extracing method 'store_hashstore_refs_files…
doulikecookiedough Sep 14, 2024
333303d
Refactor 'store_hashstore_refs_files' to catch and re-raise specific …
doulikecookiedough Sep 14, 2024
c5891d0
Add todo item in pytests for 'tag_object' and 'store_hashstore_refs_f…
doulikecookiedough Sep 14, 2024
61b320c
Format & revise '_move_and_get_checksums' docstrings to resolve linti…
doulikecookiedough Sep 16, 2024
576ee39
Add new function '_untag_object' with docstring
doulikecookiedough Sep 16, 2024
551d154
Add new custom exception 'IdentifierNotLocked', add new sync function…
doulikecookiedough Sep 16, 2024
f0881a4
Rename 'CidRefsDoesNotExist' custom exception to 'OrphanPidRefsFileFo…
doulikecookiedough Sep 16, 2024
7f74762
Add new functions '_check_object_locked_cids' and '_validate_and_chec…
doulikecookiedough Sep 16, 2024
0c8138e
Make 'store_hashstore_refs_files' private and fill out main flow of '…
doulikecookiedough Sep 16, 2024
070194d
Revise '_store_hashstore_refs_files' function signature order & updat…
doulikecookiedough Sep 16, 2024
da2e82c
Extract 'tag_object' pytests and relevant todo items from 'references…
doulikecookiedough Sep 16, 2024
a9c2f17
Extract remaining pytests from 'references' test module into 'filehas…
doulikecookiedough Sep 16, 2024
341417a
Extract pytests from 'stream' test module into 'filehashstore' test m…
doulikecookiedough Sep 16, 2024
3e482fc
Rename pytests related to retrieving hashstore entity paths for accur…
doulikecookiedough Sep 16, 2024
efa0989
Resolve todo item in '_store_hashstore_refs_files' to call '_untag_ob…
doulikecookiedough Sep 16, 2024
440b3ef
Add pytests for '_validate_and_check_cid_lock' function
doulikecookiedough Sep 16, 2024
be27311
Extract '_mark_pid_refs_file_for_deletion' function (which swallows a…
doulikecookiedough Sep 16, 2024
5ba1228
Add new function '_remove_pid_and_handle_cid_refs_deletion' with pyte…
doulikecookiedough Sep 16, 2024
0c28e4e
Remove redundant return statements
doulikecookiedough Sep 16, 2024
766d21d
Create new function '_delete_marked_files', refactor FileHashStore re…
doulikecookiedough Sep 17, 2024
8ed6638
Re-organize functions in 'filehashstore'
doulikecookiedough Sep 17, 2024
fa55ab5
Add new static method '_read_small_file_content' and refactor 'fileha…
doulikecookiedough Sep 17, 2024
8dfd5ae
Resolve todo item in '_untag_object' for 'OrphanPidRefsFileFound', ad…
doulikecookiedough Sep 17, 2024
18440a0
Add '_untag_object' pytests to resolve todo items in test module
doulikecookiedough Sep 17, 2024
a998055
Resolve todo item in '_untag_object' for 'RefsFileExistsButCidObjMiss…
doulikecookiedough Sep 17, 2024
0a15f88
Resolve todo item in '_untag_objet' for 'PidNotFoundInCidRefsFile' an…
doulikecookiedough Sep 17, 2024
8403bf6
Resolve todo item in '_untag_object' for PidRefsDoesNotExist and add …
doulikecookiedough Sep 17, 2024
4a1ab3d
Update 'README.md' to resolve linting warnings and move 'filehashstor…
doulikecookiedough Sep 18, 2024
93cce0f
Merge pull request #134 from DataONEorg/feature-122-untagobject
doulikecookiedough Sep 18, 2024
65f4eeb
Update version number and authors in 'pyproject.toml'
doulikecookiedough Sep 20, 2024
a08a6b1
Review and clean up 'filehashstore_interface' test module for 'store_…
doulikecookiedough Sep 20, 2024
2e67b39
Re-organize 'delete_if_invalid_object' in 'filehashstore' module to m…
doulikecookiedough Sep 20, 2024
7ff4722
Re-organize and clean up 'delete_if_invalid_object' method pytests in…
doulikecookiedough Sep 20, 2024
73226da
Review and clean-up 'store_metadata' pytests
doulikecookiedough Sep 20, 2024
742eb88
Review and clean-up 'retrieve_object' pytests
doulikecookiedough Sep 20, 2024
6a1b072
Review and clean-up 'retrieve_metadata' pytests
doulikecookiedough Sep 20, 2024
1d5b215
Review and clean-up 'delete_object' pytests
doulikecookiedough Sep 20, 2024
cf1f11c
Review and clean-up 'delete_metadata' pytests
doulikecookiedough Sep 20, 2024
56a4315
Review and clean-up 'get_hex_digest' pytests
doulikecookiedough Sep 20, 2024
9ea9572
Fix bug in 'find_object' where sysmeta path was not returned due to i…
doulikecookiedough Sep 20, 2024
2b0346a
Re-organize 'find_object' pytests and filehashstore code placement
doulikecookiedough Sep 20, 2024
cdf377b
Cleanup 'filehashstore' pytests module part 1 via reorganization and …
doulikecookiedough Sep 20, 2024
b9d216c
Cleanup remaining 'filehashstore' pytests module for core methods
doulikecookiedough Sep 20, 2024
1582505
Cleanuo 'filehashstore' utility and supporting methods pytests part 1…
doulikecookiedough Sep 20, 2024
d58d5e8
Revise exception messaging in '_verify_object_information', review py…
doulikecookiedough Sep 23, 2024
d49cdfa
Refactor private '_delete' method and clean-up remaining pytests
doulikecookiedough Sep 24, 2024
3ab2309
Revert 'pyproject.toml' version and add version to 'hashstore/__init_…
doulikecookiedough Sep 24, 2024
22624db
Add poetry bump version code to 'pyproject.toml'
doulikecookiedough Sep 24, 2024
50b8f8d
Update version to '1.1.0' via 'poetry version minor'
doulikecookiedough Sep 24, 2024
e780711
Update 'README.md'
doulikecookiedough Sep 24, 2024
44b1f79
Add type hints to all method signatures, fix minor issues related to …
doulikecookiedough Sep 24, 2024
b4b6e82
Merge pull request #142 from DataONEorg/feature-133-release-prep
doulikecookiedough Sep 24, 2024
b9a5635
Clean-up creation of paths in 'filehashstore' init process and revise…
doulikecookiedough Sep 24, 2024
ff29ec1
Clean-up docstrings for '_load_properties' and '_write_properties' in…
doulikecookiedough Sep 24, 2024
d3a82a9
Refactor code to replace string concatenation for creating paths with…
doulikecookiedough Sep 24, 2024
dccb260
Clean-up usage of string casting in 'delete_object'
doulikecookiedough Sep 24, 2024
91e3099
Remove redundant usage of 'rtype' in docstrings since we have added t…
doulikecookiedough Sep 24, 2024
b44662b
Fix incorrect usage of 'os.path.exists' when checking for files by us…
doulikecookiedough Sep 24, 2024
db5b8cd
Rename variables to resolve some linting typo warnings
doulikecookiedough Sep 24, 2024
f060c3e
Update 'Stream' class constructor doc string to also include 'Path' a…
doulikecookiedough Sep 25, 2024
6daa01f
Clean-up linting warnings in 'hashstoreclient' and 'filehashstore' py…
doulikecookiedough Sep 25, 2024
27b7060
Resolve linting warning for intentional pytest with 'noinspection PyT…
doulikecookiedough Sep 25, 2024
695e837
Merge pull request #143 from DataONEorg/feature-139-path-construction
doulikecookiedough Sep 25, 2024
efd3cb5
Refactor '_build_hashstore_yaml_string' to create yaml content portio…
doulikecookiedough Sep 26, 2024
d5e2341
Merge pull request #144 from DataONEorg/feature-138-yaml-lib
doulikecookiedough Sep 26, 2024
d35e9ac
Refactor the init process by declaring paralellization variables afte…
doulikecookiedough Sep 26, 2024
dedd3a5
Begin clean up logging statements by adding new method '_get_logger' …
doulikecookiedough Sep 26, 2024
71827f5
Clean up code by renaming all 'exception_message' variables with 'err…
doulikecookiedough Sep 26, 2024
6180301
Clean-up logging in 'filehashstore' init related methods and delete r…
doulikecookiedough Sep 26, 2024
692703b
Clean-up logging in 'filehashstore' interface methods and add missing…
doulikecookiedough Sep 26, 2024
47f6556
Fix bug in 'delete_object' where synchronization call was not made in…
doulikecookiedough Sep 26, 2024
f70243e
Fix typo in logging message in '_synchronize_object_locked_pids'
doulikecookiedough Sep 27, 2024
276c6e7
Clean-up logging in 'filehashstore' supporting and core methods part. 1
doulikecookiedough Sep 27, 2024
d6c3c69
Refactor '_delete_object_only', add missing logging statements and fi…
doulikecookiedough Sep 27, 2024
dbd57de
Clean-up logging in 'filehashstore' remaining supporting and core met…
doulikecookiedough Sep 27, 2024
6256bcb
Refactor and fix bug in 'store_object' where sync method was not with…
doulikecookiedough Sep 27, 2024
fe5e7db
Clean-up code/resolve linting errors for missed unused variables and …
doulikecookiedough Oct 1, 2024
0aa07b6
Merge pull request #145 from DataONEorg/feature-140-logging-enhancement
doulikecookiedough Oct 1, 2024
070c338
Update README.md with new citation section and placeholders for DOI i…
doulikecookiedough Oct 1, 2024
cf02c90
Update 'README.md' with DOI
doulikecookiedough Oct 1, 2024
6552748
Update PUBLIC API section in 'README.md'
doulikecookiedough Oct 1, 2024
a975ea0
Merge pull request #146 from DataONEorg/feature-108-citation-update-2
doulikecookiedough Oct 1, 2024
5169cd4
Resolve linting warning due to conflicting variable name 'store' in h…
doulikecookiedough Oct 1, 2024
c9e911c
Review and update 'README.md'
doulikecookiedough Oct 2, 2024
4b491d0
Minor formatting fix to 'README.md' DOI underneath version
doulikecookiedough Oct 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ venv/
ENV/
env.bak/
venv.bak/
.idea

# Spyder project settings
.spyderproject
Expand Down
252 changes: 223 additions & 29 deletions README.md

Large diffs are not rendered by default.

7 changes: 5 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[tool.poetry]
name = "hashstore"
version = "1.0.0"
version = "1.1.0"
description = "HashStore, a hash-based object store for data packages."
authors = ["Matt Jones <gitcode@magisa.org>", "Dou Mok <douming.mok@gmail.com>"]
authors = ["Dou Mok <douming.mok@gmail.com>", "Matt Jones <gitcode@magisa.org>"]
readme = "README.md"

[tool.poetry.dependencies]
Expand All @@ -16,6 +16,9 @@ black = "^22.10.0"
pylint = "^2.17.4"
pg8000 = "^1.29.8"

[tool.poetry.scripts]
hashstore = "hashstore.hashstoreclient:main"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
2,350 changes: 1,677 additions & 673 deletions src/hashstore/filehashstore.py

Large diffs are not rendered by default.

116 changes: 116 additions & 0 deletions src/hashstore/filehashstore_exceptions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
"""FileHashStore custom exception module."""


class CidRefsContentError(Exception):
"""Custom exception thrown when verifying reference files and a cid refs
file does not have a pid that is expected to be found."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class CidRefsFileNotFound(Exception):
"""Custom exception thrown when verifying reference files and a cid refs
file is not found."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class CidRefsDoesNotExist(Exception):
"""Custom exception thrown when a cid refs file does not exist."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class PidRefsContentError(Exception):
"""Custom exception thrown when verifying reference files and a pid refs
file does not contain the cid that it is expected."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class PidRefsFileNotFound(Exception):
"""Custom exception thrown when verifying reference files and a pid refs
file is not found."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class PidAlreadyExistsError(Exception):
"""Custom exception thrown when a client calls 'tag_object' and the pid
that is being tagged is already accounted for (has a pid refs file and
is found in the cid refs file)."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class PidRefsDoesNotExist(Exception):
"""Custom exception thrown when a pid refs file does not exist."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class PidNotFoundInCidRefsFile(Exception):
"""Custom exception thrown when pid reference file exists with a cid, but
the respective cid reference file does not contain the pid."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class NonMatchingObjSize(Exception):
"""Custom exception thrown when verifying an object and the expected file size
does not match what has been calculated."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class NonMatchingChecksum(Exception):
"""Custom exception thrown when verifying an object and the expected checksum
does not match what has been calculated."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class RefsFileExistsButCidObjMissing(Exception):
"""Custom exception thrown when pid and cid refs file exists, but the
cid object does not."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class HashStoreRefsAlreadyExists(Exception):
"""Custom exception thrown when called to tag an object that is already tagged appropriately."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors


class UnsupportedAlgorithm(Exception):
"""Custom exception thrown when a given algorithm is not supported in HashStore for
calculating hashes/checksums, as the default store algo and/or other operations."""

def __init__(self, message, errors=None):
super().__init__(message)
self.errors = errors
Loading
Loading