Skip to content

Commit 3f27250

Browse files
author
Sandra Mierz
authored
Merge pull request #19 from Project-TAPIR/develop
* include notebooks for OpenAIRE person-works and work-projects * include ROR ID in ORCID search * update docs
2 parents 22f05b1 + 13a19b7 commit 3f27250

8 files changed

+444
-15
lines changed

README.md

+8-2
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
[![DOI](https://zenodo.org/badge/447263093.svg)](https://zenodo.org/badge/latestdoi/447263093)
44
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/Project-TAPIR/pidgraph-notebooks/main)
55

6-
A collection of Jupyter notebooks with examples of querying different PID providers like [ORCID](https://orcid.org/), [ROR](https://ror.readme.io/), [Crossref](https://www.crossref.org/) and PID graphs like the [FREYA PID Graph](https://blog.datacite.org/powering-the-pid-graph/) and [OpenAlex](https://openalex.org/about) for connected objects.
6+
A collection of Jupyter notebooks with examples of querying different PID providers like [ORCID](https://orcid.org/), [ROR](https://ror.readme.io/), [Crossref](https://www.crossref.org/) and PID graphs like the [FREYA PID Graph](https://blog.datacite.org/powering-the-pid-graph/), [OpenAlex](https://openalex.org/about) and [OpenAIRE](https://www.openaire.eu/) for connected objects.
77

88
Currently included connections:
99
* organization-organization
@@ -17,7 +17,11 @@ Currently included connections:
1717
* person-works
1818
* input: ORCID
1919
* output: list of works authored/created by the person, each identified by their DOI
20-
* data sources: Crossref, FREYA PID Graph, OpenAlex, ORCID
20+
* data sources: Crossref, FREYA PID Graph, OpenAlex, ORCID, OpenAIRE
21+
* work-projects
22+
* input: DOI
23+
* output: list of projects the work was produced in, each identified by their OpenAIRE project ID
24+
* data sources: OpenAIRE
2125

2226

2327
Please navigate into the respective folder to see the list of available notebooks.
@@ -35,3 +39,5 @@ you can use this link to launch the notebooks on Binder where you can execute an
3539
In the joint project [TAPIR](https://projects.tib.eu/tapir/en/) (Partially Automated Persistent Identifier-based Reporting), partially automated procedures for research reporting are being tested in the context of university and non-university research. To this end, the question is being investigated :
3640

3741
To what extent can the necessary data aggregation be carried out on the basis of openly available research information using persistent identifiers?
42+
43+
*More information in our blog post "[Project TAPIR: Harvesting the power of PIDs](https://blogs.tib.eu/wp/tib/2022/03/01/project-tapir-harvesting-the-power-of-pids/)"*

organization-people/orcid_get_people_by_organization.ipynb

+13-10
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
"source": [
99
"### Query ORCID for people affiliated with an organization and filter for current employees only\n",
1010
"\n",
11-
"This notebook queries the [ORCID API](https://api.orcid.org/v3.0/) for all [people affiliated with an organization](https://info.orcid.org/faq/how-do-i-find-orcid-record-holders-at-my-institution/) and additionally narrows down the affiliation to people **currently employed** by the organization. From the resulting list of people we output the ORCID iDs.\n",
11+
"This notebook queries the [ORCID Public API](https://api.orcid.org/v3.0/) for all [people affiliated with an organization](https://info.orcid.org/faq/how-do-i-find-orcid-record-holders-at-my-institution/) and additionally narrows down the affiliation to people **currently employed** by the organization. From the resulting list of people we output the ORCID iDs.\n",
1212
"\n",
1313
"*Disclosure:\n",
1414
"The process of querying the ROR API for additional identifiers and using them to query the ORCID API for affiliated people is the same as used by the [FREYA PID Graph](https://blog.datacite.org/powering-the-pid-graph/) and is implemented in [DataCite Application API](https://doi.org/10.5438/8gb0-v673).*"
@@ -105,6 +105,7 @@
105105
"\n",
106106
"#--- example execution\n",
107107
"ror_data=query_ror_api(example_ror)\n",
108+
"organization_ror_id=example_ror.replace(\"https://ror.org/\", \"\")\n",
108109
"# if you want to see the retrieved metadata, uncomment next lines\n",
109110
"#import pprint\n",
110111
"#pprint.pprint(ror_data)"
@@ -148,7 +149,7 @@
148149
"name": "stdout",
149150
"output_type": "stream",
150151
"text": [
151-
"grid ID: grid.461819.3\n",
152+
"Grid ID: grid.461819.3\n",
152153
"Wikidata ID: Q2399120\n"
153154
]
154155
}
@@ -169,7 +170,7 @@
169170
"\n",
170171
"#--- example execution\n",
171172
"organization_grid_id=extract_grid_from_ror_data(ror_data)\n",
172-
"print(\"grid ID: \" + str(organization_grid_id or ''))\n",
173+
"print(\"Grid ID: \" + str(organization_grid_id or ''))\n",
173174
"organization_wikidata_id=extract_wikidata_from_ror_data(ror_data)\n",
174175
"print(\"Wikidata ID: \" + str(organization_wikidata_id or ''))"
175176
]
@@ -261,7 +262,7 @@
261262
},
262263
"source": [
263264
"### Connection organization -> people\n",
264-
"The second part of the process is to query for the people affiliated with the organization. For this we use the ORCID API and search for people affiliated with an organization like it is explained in the ORCID tutorial [\"How do I find ORCID record holders at my institution?\"](https://info.orcid.org/faq/how-do-i-find-orcid-record-holders-at-my-institution/). As parameters for the query we use the Grid ID and Ringgold ID for the organization.\n"
265+
"The second part of the process is to query for the people affiliated with the organization. For this we use the ORCID API and search for people affiliated with an organization like it is explained in the ORCID tutorial [\"How do I find ORCID record holders at my institution?\"](https://info.orcid.org/faq/how-do-i-find-orcid-record-holders-at-my-institution/). As parameters for the query we use the ROR ID, Grid ID and Ringgold ID for the organization.\n"
265266
]
266267
},
267268
{
@@ -391,11 +392,13 @@
391392
"# URL for ORCID search API\n",
392393
"ORCID_SEARCH_API = \"https://pub.orcid.org/v3.0/expanded-search/\"\n",
393394
"\n",
394-
"# query ORCID with an organization's Grid ID and Ringgold\n",
395-
"def query_orcid_for_affiliations(grid_id, ringgold_id):\n",
396-
" query = f\"grid-org-id:{grid_id}\" if grid_id else \"\"\n",
397-
" query += \" OR \" if grid_id and ringgold_id else \"\"\n",
398-
" query += f\"ringgold-org-id:{ringgold_id}\" if ringgold_id else \"\"\n",
395+
"# query ORCID with an organization's ROR, Grid and Ringgold ID\n",
396+
"def query_orcid_for_affiliations(ror_id, grid_id, ringgold_id):\n",
397+
" grid_search = f\"grid-org-id:{grid_id}\" if grid_id else \"\"\n",
398+
" ringgold_search = f\"ringgold-org-id:{ringgold_id}\" if ringgold_id else \"\"\n",
399+
" ror_search = f\"ror-org-id:{ror_id}\" if ror_id else \"\"\n",
400+
" orga_search_ids = [ror_search, grid_search, ringgold_search]\n",
401+
" query = ' OR '.join(filter(None, orga_search_ids))\n",
399402
"\n",
400403
" response = requests.get(url=ORCID_SEARCH_API,\n",
401404
" params={'q': query},\n",
@@ -415,7 +418,7 @@
415418
"\n",
416419
"\n",
417420
"#-- example execution\n",
418-
"affiliated_people = query_orcid_for_affiliations(organization_grid_id, organization_ringgold_id)\n",
421+
"affiliated_people = query_orcid_for_affiliations(organization_ror_id, organization_grid_id, organization_ringgold_id)\n",
419422
"affiliated_count = affiliated_people.get('num-found','')\n",
420423
"print(f\"Number of affiliated people: {affiliated_count}\")\n",
421424
"\n",

person-works/README.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,5 @@ Currently available PID Graphs:
66
* [Crossref](https://www.crossref.org/)
77
* [FREYA PID Graph](https://blog.datacite.org/powering-the-pid-graph/)
88
* [OpenAlex](https://openalex.org/about)
9-
* [ORCID](https://orcid.org/)
9+
* [ORCID](https://orcid.org/)
10+
* [OpenAIRE](https://www.openaire.eu/)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"### Query OpenAIRE for publications authored by a person\n",
8+
"This notebook queries the [OpenAIRE HTTP API](https://graph.openaire.eu/develop/api.html) via its `/publications` endpoint for publications authored by a person. It takes an ORCID iD as input which is used to filter for publications where one of the creators' `orcid` field matches the given ORCID iD. From the resulting list of publications we output all DOIs.\n",
9+
"\n",
10+
"*Note:\n",
11+
"The API has several different endpoints for research outputs: they are divided into publications, research data, software metadata and other research products, so to get a full picture about a person's research output, you would have to query all of these endpoints and union their results.*"
12+
]
13+
},
14+
{
15+
"cell_type": "code",
16+
"execution_count": 1,
17+
"metadata": {
18+
"pycharm": {
19+
"name": "#%%\n"
20+
}
21+
},
22+
"outputs": [],
23+
"source": [
24+
"# Prerequisites:\n",
25+
"import requests # dependency for making HTTP calls\n",
26+
"from benedict import benedict # dependency for dealing with json"
27+
]
28+
},
29+
{
30+
"cell_type": "markdown",
31+
"metadata": {
32+
"collapsed": true,
33+
"pycharm": {
34+
"name": "#%% md\n"
35+
}
36+
},
37+
"source": [
38+
"The input for this notebook is an ORCID iD, e.g. '`0000-0003-2499-7741`'."
39+
]
40+
},
41+
{
42+
"cell_type": "code",
43+
"execution_count": 2,
44+
"metadata": {
45+
"pycharm": {
46+
"name": "#%%\n"
47+
}
48+
},
49+
"outputs": [],
50+
"source": [
51+
"# input parameter\n",
52+
"example_orcid_id=\"0000-0003-2499-7741\""
53+
]
54+
},
55+
{
56+
"cell_type": "markdown",
57+
"metadata": {},
58+
"source": [
59+
"We use it to query the OpenAIRE HTTP API for publications that specified the ORCID iD within their metadata in one of the creators `orcid` field. Since the API uses pagination, we need to loop through all pages to get the complete result set."
60+
]
61+
},
62+
{
63+
"cell_type": "code",
64+
"execution_count": 3,
65+
"metadata": {
66+
"pycharm": {
67+
"name": "#%%\n"
68+
}
69+
},
70+
"outputs": [],
71+
"source": [
72+
"# OpenAIRE endpoint to query for publications\n",
73+
"OPENAIRE_API_PUBLICATIONS = \"https://api.openaire.eu/search/publications\"\n",
74+
"\n",
75+
"# query OpenAIRE for all publications that are connected to orcid\n",
76+
"def query_openaire_for_person2publications(orcid_id):\n",
77+
" page = 1\n",
78+
" max_page = 1\n",
79+
"\n",
80+
" while page <= max_page:\n",
81+
" params = {'orcid': orcid_id, 'page': page, 'format': \"json\"}\n",
82+
" response = requests.get(url=OPENAIRE_API_PUBLICATIONS,\n",
83+
" params=params)\n",
84+
" response.raise_for_status()\n",
85+
" result=response.json()\n",
86+
"\n",
87+
" # calculate max page number in first loop\n",
88+
" if max_page == 1:\n",
89+
" max_page = determine_max_page(result)\n",
90+
" page = page + 1\n",
91+
" yield result\n",
92+
"\n",
93+
"# calculate max number of result pages\n",
94+
"def determine_max_page(response_data):\n",
95+
" response_dict = benedict.from_json(response_data)\n",
96+
" items_total = response_dict.get('response.header.total.$')\n",
97+
" items_per_page = response_dict.get('response.header.size.$')\n",
98+
" max_page_ceil = items_total // items_per_page + bool(items_total % items_per_page)\n",
99+
" return max_page_ceil\n",
100+
"\n",
101+
"\n",
102+
"# ---- example execution\n",
103+
"list_of_pages=query_openaire_for_person2publications(example_orcid_id)"
104+
]
105+
},
106+
{
107+
"cell_type": "markdown",
108+
"metadata": {},
109+
"source": [
110+
"From the resulting list of publications we extract and print out each title and DOI. \n",
111+
"\n",
112+
"*Note: publications that do not have a DOI assigned, will not be printed.*"
113+
]
114+
},
115+
{
116+
"cell_type": "code",
117+
"execution_count": 4,
118+
"metadata": {},
119+
"outputs": [
120+
{
121+
"name": "stdout",
122+
"output_type": "stream",
123+
"text": [
124+
"Number of publications found: 6\n",
125+
"\n",
126+
"10.15488/11463, Roadmap to FAIR Research Information in Open Infrastructures\n",
127+
"10.1515/bd.2006.40.4.466, Informationsvermittlung: Personalisiertes Lernen in der Bibliothek: das Düsseldorfer Online-Tutorial (DOT) Informationskompetenz\n",
128+
"10.1080/00048623.2006.10755322, Teaching Information Literacy with the Lerninformationssystem\n",
129+
"10.3389/frma.2021.694307, Enhancing Knowledge Graph Extraction and Validation From Scholarly Publications Using Bibliographic Metadata\n",
130+
"10.3897/rio.7.e66264, OPTIMETA – Strengthening the Open Access publishing system through open citations and spatiotemporal metadata\n",
131+
"10.1016/j.procs.2019.01.074, The Research Core Dataset (KDSF) in the Linked Data context\n"
132+
]
133+
}
134+
],
135+
"source": [
136+
"# from the result pages, extract the data about each publication\n",
137+
"def extract_publications_from_page(page):\n",
138+
" return [pub for pub in benedict.from_json(page).get('response.results.result') or []]\n",
139+
"\n",
140+
"# extract DOI from publication\n",
141+
"def extract_doi(pub):\n",
142+
" oaf_result=benedict.from_json(pub).get('metadata.oaf:entity.oaf:result')\n",
143+
"\n",
144+
" # unfortunately the json data is inconsistently modeled:\n",
145+
" # if there is one pid/title for a publication, it is a json object\n",
146+
" # if there are multiple pids/titles for a publication, they form a json list\n",
147+
" pids=oaf_result.get('pid') or []\n",
148+
" is_doi = lambda pid: pid.get('@classid')==\"doi\"\n",
149+
" if isinstance(pids, list):\n",
150+
" dois=[pid['$'] for pid in pids if is_doi(pid)]\n",
151+
" else:\n",
152+
" dois= [pids['$']] if is_doi(pids) else []\n",
153+
" doi=dois[0] if dois else None # pick the first one\n",
154+
" \n",
155+
" titles=oaf_result.get('title') or []\n",
156+
" is_main_title = lambda title: title.get('@classid')==\"main title\"\n",
157+
" if isinstance(titles, list):\n",
158+
" main_titles=[title['$'] for title in titles if is_main_title(title)]\n",
159+
" else:\n",
160+
" main_titles=[titles['$']] if is_main_title(titles) else []\n",
161+
" title=main_titles[0] if main_titles else None # pick the first one\n",
162+
"\n",
163+
" return doi, title\n",
164+
"\n",
165+
"\n",
166+
"#--- example execution\n",
167+
"for page in list_of_pages or []:\n",
168+
" publications=extract_publications_from_page(page)\n",
169+
" print(f\"Number of publications found: {len(publications)}\\n\")\n",
170+
" for pub in publications:\n",
171+
" doi,title = extract_doi(pub)\n",
172+
" if doi:\n",
173+
" print(f\"{doi}, {title}\")"
174+
]
175+
}
176+
],
177+
"metadata": {
178+
"kernelspec": {
179+
"display_name": "Python 3 (ipykernel)",
180+
"language": "python",
181+
"name": "python3"
182+
},
183+
"language_info": {
184+
"codemirror_mode": {
185+
"name": "ipython",
186+
"version": 3
187+
},
188+
"file_extension": ".py",
189+
"mimetype": "text/x-python",
190+
"name": "python",
191+
"nbconvert_exporter": "python",
192+
"pygments_lexer": "ipython3",
193+
"version": "3.9.6"
194+
}
195+
},
196+
"nbformat": 4,
197+
"nbformat_minor": 1
198+
}

person-works/openalex_get_works_by_person.ipynb

+1-1
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@
4444
"id": "Rt7GUbFcaNxi"
4545
},
4646
"source": [
47-
"The input for the query is an ORCID URL, e.g. '`https://orcid.org/0000-0003-2499-7741`'"
47+
"The input for this notebook is an ORCID URL, e.g. '`https://orcid.org/0000-0003-2499-7741`'"
4848
]
4949
},
5050
{

person-works/orcid_get_works_by_person.ipynb

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
"source": [
99
"### Query ORCID for works authored by a person\n",
1010
"\n",
11-
"This notebook queries the [ORCID API](https://pub.orcid.org/v3.0/) to retrieve works listed in a person's ORCID record. It takes an ORCID URL or iD as input to retrieve the ORCID record of a person and the works listed on it. From the resulting list of works we output all DOIs."
11+
"This notebook queries the [ORCID Public API](https://pub.orcid.org/v3.0/) to retrieve works listed in a person's ORCID record. It takes an ORCID URL or iD as input to retrieve the ORCID record of a person and the works listed on it. From the resulting list of works we output all DOIs."
1212
]
1313
},
1414
{

work-projects/README.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
## work-projects
2+
3+
A Jupyter notebook showing an example of using a persistent identifier for a publication (DOI)
4+
as input for retrieving the project a work was produced in (identified by its OpenAIRE project ID).
5+
6+
* [OpenAIRE](https://www.openaire.eu/)

0 commit comments

Comments
 (0)