Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Materialize mongo alldocs collection #550

Merged
merged 16 commits into from
Jul 10, 2024
Merged

Conversation

dwinston
Copy link
Collaborator

@dwinston dwinston commented Jun 7, 2024

Description

Fixes #543

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How Has This Been Tested?

  • Tested manually in local instance of dagster
    image

  • added test_materialize_all_docs()

Configuration Details: none

Checklist:

  • My code follows the style guidelines of this project (have you run black nmdc_runtime/?)
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (in docs/ and in https://github.com/microbiomedata/NMDC_documentation/?)
  • I have added tests that prove my fix is effective or that my feature works, incl. considering downstream usage (e.g. https://github.com/microbiomedata/notebook_hackathons) if applicable.
  • New and existing unit and functional tests pass locally with my changes (make up-test && make test-run)

@PeopleMakeCulture PeopleMakeCulture changed the title prototype alldocs job [DRAFT - DO NOT MERGE] prototype alldocs job Jun 7, 2024
@PeopleMakeCulture PeopleMakeCulture changed the title [DRAFT - DO NOT MERGE] prototype alldocs job Create dagster op to re[materialize] mongo alldocs collection Jun 19, 2024
@PeopleMakeCulture PeopleMakeCulture self-assigned this Jun 20, 2024
@PeopleMakeCulture
Copy link
Collaborator

PeopleMakeCulture commented Jun 20, 2024

TODO:

  • standardize alldocs vs all_docs usage
  • update documentation in notebook
  • consider refactoring notebook to use op.py code and utils

@PeopleMakeCulture PeopleMakeCulture self-requested a review June 20, 2024 20:37
@PeopleMakeCulture PeopleMakeCulture marked this pull request as ready for review June 28, 2024 18:33
@sujaypatil96
Copy link
Collaborator

It looks like there are a couple of merge conflicts that need to be resolved on a couple of files in this PR, would you like to resolve them @PeopleMakeCulture, or should I do it? they seem fairly trivial so I'm happy to do it as well, do let me know.

@PeopleMakeCulture
Copy link
Collaborator

It looks like there are a couple of merge conflicts that need to be resolved on a couple of files in this PR, would you like to resolve them @PeopleMakeCulture, or should I do it? they seem fairly trivial so I'm happy to do it as well, do let me know.

@sujaypatil96 Hey sorry for the delayed response. I'm still catching up from covid last week. Could you go ahead and resolve the conflicts? Happy to take a look if there's anything confusing

@PeopleMakeCulture
Copy link
Collaborator

Dependency versions updated via: #585

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Materialize a version of all_docs collection in dev mongodb
3 participants