Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding a metadata flag for signaling successful ingestion #140

Conversation

ktsitsi
Copy link
Collaborator

@ktsitsi ktsitsi commented Oct 23, 2024

This PR:

  • Adds a metadata KV at a group level of an ingested image signalling if the ingestion was completed successfully after all the steps.
  • The metadata key valid gets a boolean value of either True or False
  • During every ingestion a check is being performed to find if the desination path already exists and is a group. In this scenario we check if the group's metadata contain the key valid. If not this signals a corrupted ingested file (possibly from a previous run) and the response to that is for the group to be overwritten from scratch.
  • In case the group exists but the ingestion runs in order for more layers to be added into the image incremental ingestion the flag is not enabled. We initially remove again the valid KV from the group's metadata until re-instantiate at the end of the current ingestion that is being running.

@@ -410,6 +410,26 @@ def to_tiledb(
metadata = {}
original_metadata = {}

# Checks group validity
overwrite = False
Copy link
Contributor

@spencerseale spencerseale Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should overwrite be an arg that defaults to False? That would give us flexibility to overwrite every image. Although uncommon, may be times when we need to force overwrite. Could then make that arg accessible from tiledb.cloud.bioimage.ingest as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The caveat would be of course that if an image is invalid it will still be overwritten even if overwrite=False set in the caller

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeap the direction of this patch is to mitigate retrial issues and corrupted leftovers, that the user fails to recognize. If users really want to overwrite/re-ingest an image on purpose they can always do it by changing the destination path and manually deleting the previous record.

@ktsitsi ktsitsi merged commit 3eb9388 into main Oct 24, 2024
4 checks passed
@ktsitsi ktsitsi deleted the kt/sc-58162/explicit-ack-metadata-kv-in-bioimg-groups-flagging-successful-ingestion branch October 24, 2024 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants