Submission portal metadata ingest #118

JamesTessmer · 2023-03-13T21:32:58Z

Code for ingesting submission portal metadata -> mongo.

I used Sujay's gold-bioscales branch as a base as we had discussed needing to use the RuntimeApiSiteClient.
So far the code includes:

API call to collect json output from the submission portal
Processing of samples in that response into floats/textValues/quanitityValues/etc as needed, including minting new IDs
Putting these into an NMDC database object and then dumping that as a json.

Additions for this task include an example driver file (main.py), a python file that has the submission_portal_client object and helper functions (submission_portal_client.py).

Initially a CSV file mapping the submission portal metadata slots to the MongoDB slots was needed, but with the updated to the submission portal these now match and it has been removed.

sujaypatil96 · 2023-03-29T18:08:51Z

sample_annotator/clients/submission_portal_client.py

+#independent functions that I've added to this overarching class
+
+#API things are still missing as I only just got access to them ~1 hour before pushing this code up
+class submission_portal_client:


Class names should typically be PascalCase. So you would have SubmissionPortalClient.

Ive made this update in the code.

sujaypatil96 · 2023-03-29T18:17:17Z

sample_annotator/clients/submission_portal_client.py

+
+
+    def mint_nmdc_ids(self,num_ids):
+        json_dic = {"schema_class": {"id": "nmdc:Activity"},"how_many": num_ids}


Are you sure you want to be minting ids for the nmdc:Activity type? Isn't it nmdc:Biosample because you're trying to assign ids to each row from the Submission Portal interface, and each of those rows are biosamples?

I was copying the example for minting IDs and hadnt realized there were multiple types. Ive made this change and it now mints biosample IDs.

sujaypatil96 · 2023-03-29T18:21:21Z

sample_annotator/main.py

+import sys
+from clients.submission_portal_client import submission_portal_client
+
+
+
+client = submission_portal_client('.env')
+r = client.get_submission_json('e4cf7c2c-29b4-4924-8452-2bd3c8e518db')
+biosample_db_lis = client.process_json_response(r)
+for i in range(0,len(biosample_db_lis)):
+    client.dump_db(biosample_db_lis[i],"output_" + str(i) + ".json")


Make this a test for your submission_portal_client.py code instead of a Python module this way?

Main.py is removed and a unit test file has been created in the tests directory with a few tests.

sujaypatil96 · 2023-03-29T18:33:35Z

sample_annotator/clients/submission_portal_client.py

+                param_dic = {}
+                for key in samp_dic.keys():
+
+                    if(samp_dic[key] == 'nan' or samp_dic[key] == 'null' or samp_dic[key] == None):


Any reason why you're using parentheses in your if conditionals? It's not required if you're not trying to enforce an order of operations. If you had a combination of or, and and () groupings it would be useful to wrap if conditionals in parentheses, but I don't think it's necessary here.

Theres no reason other than habit. I also work with a few other languages that do require parentheses, so I got into the habit of always using them. I can remove them if needed.

sujaypatil96 · 2023-03-29T18:36:58Z

sample_annotator/clients/submission_portal_client.py

+
+                    if (key in string_params):
+                        if (key == 'id'):
+                            param_dic['alternative_identifiers'] = samp_dic[key] #this vlue might need to be different than 'alternative_identifiers'


Yup, as you correctly anticipated, it might need to be source_mat_id

Ive made this change, checking the updated submission portal API output this field is now labeled as source_mat_id there as well.

sujaypatil96 · 2023-03-29T18:40:30Z

sample_annotator/clients/submission_portal_client.py

+
+                        else:
+                            if(";" in samp_dic[key]):
+                                temp = samp_dic[key].split(";")


The split() operation returns a list. Did you mean to pick the first element and use that as the id after you've done the split?

There were a few issues with fields that had semicolon delimiters so if they were present i was converting them to a list. With the updated api output this isnt a problem anymore so ive removed this section of code.

sujaypatil96 · 2023-03-29T18:44:25Z

sample_annotator/clients/submission_portal_client.py

+                            temp = nmdc.ControlledTermValue(term=nmdc.OntologyClass(id=env_code))
+                            param_dic[key] = temp


This needs to be a ControlledIdentifiedTermValue and not a ControlledTermValue. You can what dataclasses to use for slots by looking at this page: https://microbiomedata.github.io/nmdc-schema/Biosample/

Ive made this update and the output now looks correct, including the id, name, and raw value.

sujaypatil96 · 2023-03-29T18:46:16Z

sample_annotator/clients/submission_portal_client.py

+
+                    #if something shows up in the else statement it is missing from the mappings
+                    else:
+                        print(key + " not found in any current param list")


General tip: Use logging module logger statements instead of print statements

Ive added logging to replace prints now.

sujaypatil96 · 2023-03-29T18:47:52Z

sample_annotator/clients/submission_portal_client.py

+                    temp = nmdc.ControlledTermValue(term=nmdc.OntologyClass(id="0000"))
+                    param_dic[item] = temp
+
+                sample = nmdc.Biosample(**param_dic)


Excellent 🌟🌟

sujaypatil96 · 2023-03-30T17:20:45Z

sample_annotator/clients/submission_portal_client.py

+import logging
+from clients.nmdc.runtime_api_client import RuntimeApiSiteClient
+from dotenv import dotenv_values
+import dateutil.parser as parser


from dateutil import parser

Implemented this change.

sujaypatil96 · 2023-03-30T17:21:18Z

sample_annotator/clients/submission_portal_client.py

@@ -0,0 +1,189 @@
+import nmdc_schema.nmdc as nmdc
+import linkml
+import linkml_runtime.dumpers.json_dumper as dumper


from linkml_runtime.dumpers import json_dumper

Implemented this change.

sujaypatil96 · 2023-03-30T17:22:34Z

sample_annotator/clients/submission_portal_client.py

+    #database: an NMDC database object
+    #outfile_path: a string, filepath for desired output
+    def dump_db(self,database,outfile_path):
+        dumper.dump(element = database, to_file = outfile_path)


Use indent=4 as argument to dump() for better formatted JSON.

I couldnt figure this one out, i didnt see an indent parameter for the function so this is unchanged at the moment

sujaypatil96

Left very trivial comments on the PR again @JamesTessmer. Good work!

Updated documentation and finalized some functions in sub portal client.

JamesTessmer · 2023-03-31T17:27:59Z

@sujaypatil96 just pushed up some changes with suggested modifications. This includes the water content fix.

sujaypatil96 and others added 30 commits December 15, 2022 15:27

retrieve all fields from GOLD API

955cc79

fix tests related to gold ingest pipeline

205403e

Database output compatible with v7 of the schema

4771c40

simplify compute_project_set() signature

936a806

collected_from slot populated with FieldResearchSite

0937ebb

do not populate slots with empty strings

084b35d

upgrade to v7 of NMDC Schema

d033fd9

biosample ids nmdc prefix and capture envo term labels

5351936

update EMP500 JSON output from current GOLD NMDC pipeline

54eef6a

use nmdc prefix for identifiers

797c01e

nmdc id minting service to generate biosample ids

7efff85

add .env example which needs to be in the project root

4657e2a

enable pulling of sequencing project information

5cdcdcf

updated JSON output for EMP500 study

0052df7

remove creds from .env file

a695792

capture samp_name and make ids for FieldResearchSite

605ebcc

pull in img_identifiers information from analysis project

f1f0678

update pipeline to use latest version of nmdc-schema package

1ff3256

pull in host_taxid information from GOLD

223ee11

EMP500 dataset output from transformation pipeline for test

f3363ed

use GOLD: prefix instead of gold: prefix

c6d3700

field research site ids processing and data portal enrichment

e6f8156

remove print statements

5274c7b

use correct GOLD API key to pull in host_taxid

2d3227c

update .env example to point to prod api rather than dev api

a860805

update EMP500 export in test

b13cf3d

account for multiple IMG identifiers

e55db81

Initial commit of submission portal code

0f34e06

Some changes to include API calls

e9f2b35

Changing file structure a little bit to match current structure better

36c4d18

JamesTessmer marked this pull request as draft March 13, 2023 21:35

James Tessmer added 2 commits March 24, 2023 10:55

Removing some files and relocating others to where they should be.

c6da845

added API interaction and refactored for new formatting

a7097d9

JamesTessmer marked this pull request as ready for review March 24, 2023 18:03

JamesTessmer mentioned this pull request Mar 27, 2023

Script for converting submission portal metadata -> Mongo for postgres and data portal ingest microbiomedata/issues#121

Closed

sujaypatil96 reviewed Mar 29, 2023

View reviewed changes

Addressing PR comments: date format, ENVO format, parsing quantities

7a150b7

sujaypatil96 reviewed Mar 30, 2023

View reviewed changes

James Tessmer added 2 commits March 30, 2023 14:09

Pushing for error checking. Water content issue.

a5262fd

Removing main.py, modified to be unit tests.

541883d

Updated documentation and finalized some functions in sub portal client.

pkalita-lbl mentioned this pull request Apr 19, 2023

Port submission portal ingest code from sample-annotator to nmdc-runtime microbiomedata/issues#224

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Submission portal metadata ingest #118

Submission portal metadata ingest #118

JamesTessmer commented Mar 13, 2023 •

edited

Loading

sujaypatil96 Mar 29, 2023

JamesTessmer Mar 30, 2023

sujaypatil96 Mar 29, 2023

JamesTessmer Mar 30, 2023

sujaypatil96 Mar 29, 2023

JamesTessmer Mar 31, 2023

sujaypatil96 Mar 29, 2023

JamesTessmer Mar 30, 2023

sujaypatil96 Mar 29, 2023

JamesTessmer Mar 30, 2023

sujaypatil96 Mar 29, 2023

JamesTessmer Mar 30, 2023

sujaypatil96 Mar 29, 2023

JamesTessmer Mar 30, 2023

sujaypatil96 Mar 29, 2023

JamesTessmer Mar 30, 2023

sujaypatil96 Mar 29, 2023

sujaypatil96 Mar 30, 2023

JamesTessmer Mar 31, 2023

sujaypatil96 Mar 30, 2023

JamesTessmer Mar 31, 2023

sujaypatil96 Mar 30, 2023

JamesTessmer Mar 31, 2023

sujaypatil96 left a comment

JamesTessmer commented Mar 31, 2023



		def mint_nmdc_ids(self,num_ids):
		json_dic = {"schema_class": {"id": "nmdc:Activity"},"how_many": num_ids}

		temp = nmdc.ControlledTermValue(term=nmdc.OntologyClass(id=env_code))
		param_dic[key] = temp

Submission portal metadata ingest #118

Are you sure you want to change the base?

Submission portal metadata ingest #118

Conversation

JamesTessmer commented Mar 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sujaypatil96 left a comment

Choose a reason for hiding this comment

JamesTessmer commented Mar 31, 2023

JamesTessmer commented Mar 13, 2023 •

edited

Loading