Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename fields in Strapi data (RPB-160) #81

Merged
merged 39 commits into from
Jun 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
36f3d7d
Rename `f00_` to `rpbId` in Strapi title data (RPB-160)
fsteeg Apr 29, 2024
be36408
Rename `f14_` to `doNotIndex` in Strapi title data (RPB-160)
fsteeg Apr 30, 2024
887d5c5
Rename `f20_` to `title` in Strapi title data (RPB-160)
fsteeg Apr 30, 2024
a188237
Rename `f39_` to `responsibilityStatement` in title data (RPB-160)
fsteeg Apr 30, 2024
273f837
Rename `f77_` to `extent` in title data (RPB-160)
fsteeg Apr 30, 2024
60934d4
Map `f21_` to `preferredTitle` in Strapi title data (RPB-160)
fsteeg May 2, 2024
c99cb45
Update URL label for current Strapi data model
fsteeg May 2, 2024
e347134
Map `f28_` to `alternativeSpelling` in Strapi title data (RPB-160)
fsteeg May 2, 2024
8966362
Rename fields f70b, f70h, f70j, f70s, f70z (RPB-160)
fsteeg May 2, 2024
f847db1
Rename f76a, f76b to publicationDate, publicationYear (RPB-160)
fsteeg May 2, 2024
7440079
Rename `f18_` to `inCollection` in Strapi title data (RPB-160)
fsteeg May 3, 2024
603f621
Rename `f40` to `person` with `value` & `type` subfields (RPB-160)
fsteeg May 3, 2024
10301a9
Don't create URIs for free text contribution values
fsteeg May 6, 2024
85945ca
Rename `f60` to `corporateBody` with `value` & `type` (RPB-160)
fsteeg May 6, 2024
0ff755a
Simplify contribution macros for unified `value` field (RPB-160)
fsteeg May 6, 2024
4a61a76
Rename f24_ and f27_ to parallel- and alternativeTitle (RPB-160)
fsteeg May 6, 2024
dc7d416
Rename `f81_` to `note` with `value` subfield in Strapi (RPB-160)
fsteeg May 6, 2024
156592a
Rename `f85_` to `isPartOf` w/ `value` subfield in Strapi (RPB-160)
fsteeg May 6, 2024
69a4043
Rename `f30.f30_` to `subject.value` in Strapi data (RPB-160)
fsteeg May 7, 2024
1d1deab
Rename `f32.f32_.f32_` to `subjectComponentList.subject.value`
fsteeg May 7, 2024
605de82
Rename `f31.f31_` to `spatial.value` in Strapi data (RPB-160)
fsteeg May 7, 2024
6e2b800
Rename `f70.f70_` to `bibliographicCitation.value` (RPB-160)
fsteeg May 7, 2024
bdf8b1b
Rename `f90.f90_` to `url.value`, `f90.n90` to `url.type` (RPB-160)
fsteeg May 7, 2024
34ff746
Rename `f90a.f90a` to `item.value` in Strapi title data (RPB-160)
fsteeg May 8, 2024
c869675
Rename `f36_` to `type` in Strapi independent-work data (RPB-160)
fsteeg May 13, 2024
2743d0c
Rename `f01` to `volumeIn` in Strapi independent-work (RPB-160)
fsteeg May 14, 2024
acffbde
Rename `f74_` to `publicationLocation`, `f75_` to `publishedBy`
fsteeg May 14, 2024
f1a7acf
Rename `f71_` to `edition`, include in lobid output (RPB-160)
fsteeg May 14, 2024
4aa7bb1
Add `f20u` to `isPartOf` in Strapi independent-work data (RPB-160)
fsteeg May 14, 2024
e14d19a
Rename `f86_` to `otherId` in Strapi title data (RPB-160)
fsteeg May 15, 2024
8c1d5b6
Rename `f87` to `isbn` in Strapi title data (RPB-160)
fsteeg May 15, 2024
b1a4316
Add `isbn` values to lobid title data (RPB-160)
fsteeg May 15, 2024
d52b660
Remove old fields after adding, rename & clean up macros (RPB-160)
fsteeg May 15, 2024
4efc913
Rename `f00_` to `rpbId` in RPB authority data (RPB-160)
fsteeg May 15, 2024
ed203a1
Rename `f99z` to `type` in Strapi RPB authority data (RPB-160)
fsteeg May 15, 2024
3a7e14e
Rename `f3na` to `preferredName` in Strapi authority data (RPB-160)
fsteeg May 15, 2024
5f4436b
Rename `f3nz` to `otherInformation` in authority data (RPB-160)
fsteeg May 15, 2024
e374fc2
Rename `f3n` to `variantName`, `f3r` to `relatedEntity` (RPB-160)
fsteeg May 15, 2024
4fd8064
Rename and clean up authority data macros (RPB-160)
fsteeg May 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,9 @@ This writes a single `.json` file to `output/` (it's actually JSON lines, but th
### Import strapi data

```bash
sbt "runMain rpb.ETL conf/rpb-test-titel-import.flux PICK=all_equal('f36_','u') PATH=articles"
sbt "runMain rpb.ETL conf/rpb-test-titel-import.flux PICK=all_equal('f36_','Monografie') PATH=independent-works"
sbt "runMain rpb.ETL conf/rpb-test-titel-import.flux PICK=all_equal('f36_','Band') PATH=independent-works"
sbt "runMain rpb.ETL conf/rpb-test-titel-import.flux PICK=all_equal('type','u') PATH=articles"
sbt "runMain rpb.ETL conf/rpb-test-titel-import.flux PICK=all_equal('type','Monografie') PATH=independent-works"
sbt "runMain rpb.ETL conf/rpb-test-titel-import.flux PICK=all_equal('type','Band') PATH=independent-works"
sbt "runMain rpb.ETL conf/rpb-test-titel-import.flux PICK=all_equal('f36t','MultiVolumeBook') PATH=independent-works"
```

Expand All @@ -78,9 +78,9 @@ curl --request DELETE http://localhost:1337/api/articles/[1-5]

After import they are available at e.g. https://rpb-cms-test.lobid.org/api/articles?populate=*

Entries using the same path can be filtered, e.g. to get only volumes (`f36_=sbd`):
Entries using the same path can be filtered, e.g. to get only volumes (`type=Band`):

https://rpb-cms-test.lobid.org/api/independent-works?filters[f36_][$eq]=sbd&populate=*
https://rpb-cms-test.lobid.org/api/independent-works?filters[type][$eq]=Band&populate=*

### Import SKOS data into strapi

Expand Down Expand Up @@ -132,9 +132,9 @@ This writes individual `.json` files for Strapi records to `output/`.
### Compare export data

```bash
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('f36_','u') PATH=articles"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('f36_','s') PATH=independent-works"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('f36_','sbd') PATH=independent-works"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('type','u') PATH=articles"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('type','Monografie') PATH=independent-works"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('type','Band') PATH=independent-works"
sbt "runMain rpb.ETL conf/test-export-compare-strapi.flux PICK=all_equal('f36t','MultiVolumeBook') PATH=independent-works"
```

Expand Down
1 change: 1 addition & 0 deletions conf/RPB-Export_HBZ_Titel_Test.txt
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,4 @@
[/]#00 929t920749[/]#20 Sachstand der heutigen Gew„sserbelastung : Belastung der letzten Jahrzehnte ; Verhalten von Schadstoffen in Gew„ssern ; Schadensf„lle[/]#30 _sn142382_[/]#31 _r22_[/]#32 _z64_ ; _4128022n2_[/]#35 _r22_ ; _sn142382_[/]#36 s[/]#37 _sn142382_ ; _r22_[/]#39 Hubert Hellmann[/]#40 _00Pn92k0749a_[/]#74 Koblenz[/]#75 Bundesanst. fr Gew„sserkunde[/]#76a1992[/]#76b1992[/]#77 28 Bl. : graph. Darst.[/]#86 BfG-0669[/]#90d92A/173[/]#95 ZR-19930513/09:54:51ZR[/]#96 ZR-19930513/09:54:51ZR[/]
[/]#00 121t231109[/]#20 ªDerª Wiedweg - blaues Wasser, grne Pfade : Etappenwandern von der Quelle bis zur Mndung[/]#30 _sn102070_[/]#30a_sn102070_[/]#31 _r22_[/]#31a_r10_[/]#32 _929n110124_ ; _4155569n7_[/]#35 _r22_ ; _sn102070_[/]#35a_r10_ ; _sn102070_[/]#36 u[/]#37 _sn102070_ ; _r22_[/]#37a_sn102070_ ; _r10_[/]#70b118[/]#70h3[/]#70j2023[/]#70s48-52[/]#70tªDieª Eifel[/]#76a2023[/]#76b2023[/]#77 Illustrationen[/]#88 0176-8255[/]#90cZ 1666-118(2023)[/]#95 20230927/11:44:46TR[/]#96 20230927/11:44:46TR[/]
[/]#00 107t182053[/]#20 Friedrich Christian Laukhard (1757-1822) : Schriftsteller, Radikalaufkl„rer und gelehrter Soldat[/]#30 _sn768030_[/]#31 _r99_ _o33105070_[/]#32 _118726692_ ; _4006804n3_[/]#35 _r99_ _o33105070_ ; _sn768030_[/]#36 s[/]#37 _sn768030_ ; _r99_ _o33105070_[/]#39 Guido Naschert (Hg.)[/]#41 _00Gn18s2053186a_[/]#74 Paderborn[/]#75 Ferdinand Sch”ningh[/]#76a[2017][/]#76b2017[/]#77 218 Seiten : Illustrationen, Karte[/]#83 Laukhard, Friedrich Christian / 1757-1822[/]#87 978-3-506-77967-0[/]#90b117-3134[/]#90d2020/1429[/]#90fhttp://digitale-objekte.hbz-nrw.de/storage2/2017/08/19/file_28/7343682.pdf[/]#90ghttps://www.hsozkult.de/publicationreview/id/reb-22111[/]#95 20180620/10:01:03HBZ[/]#96 20200603/09:22:21KO[/]#983HT019399122[/]
[/]#00 929t160199[/]#20 Wandern auf Rheinsteig und Rheinburgenweg[/]#21 Wandern auf dem Rheinsteig und dem Rheinburgenweg[/]#30 _sn102070_[/]#31 _r22_ _4126432n0_[/]#32 _7501137n2_ ; _7500776n9_ ; _4064532n0_[/]#35 _r22_ _4126432n0_ ; _sn102070_[/]#36 s[/]#37 _sn102070_ ; _r22_ _4126432n0_[/]#39 Manfred B”ckling[/]#40 _124591736_[/]#71 4., aktualisierte Auflage[/]#74 Ostfildern[/]#75 DuMont Reiseverlag[/]#76a2012[/]#76b2012[/]#77 180 Seiten : Illustrationen, Diagramme, Karten[/]#81 Enth„lt Index[/]#83 Boeckling, Manfred / 1970-[/]#85 DuMont aktiv[/]#87 978-3-7701-8016-5[/]#90d2016/311[/]#90fhttp://d-nb.info/1069609889/04[/]#95 20160128/13:51:41HBZ[/]#96 20160208/08:24:33ZR[/]#983HT018859168[/]
3 changes: 2 additions & 1 deletion conf/output/test-output-27.json
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,6 @@
"label" : "Autor/in"
},
"type" : [ "Contribution" ]
} ]
} ],
"edition" : [ "Nachdr. der Original-Ausg." ]
}
3 changes: 2 additions & 1 deletion conf/output/test-output-3.json
Original file line number Diff line number Diff line change
Expand Up @@ -113,5 +113,6 @@
"fulltextOnline" : [ {
"id" : "https://nbn-resolving.org/urn:nbn:de:hbz:061:2-24485",
"label" : "urn:nbn:de:hbz:061:2-24485"
} ]
} ],
"edition" : [ "[Nur Ill.]" ]
}
1 change: 1 addition & 0 deletions conf/output/test-output-30.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"type" : [ "BibliographicResource", "Article" ],
"rpbId" : "107t982370",
"id" : "https://lobid.org/resources/107t982370",
"isbn" : [ "3-89331-303-6" ],
"inCollection" : [ {
"id" : "http://lobid.org/resources/HT013494180#!",
"type" : [ "Collection" ],
Expand Down
1 change: 1 addition & 0 deletions conf/output/test-output-31.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"type" : [ "BibliographicResource", "Article" ],
"rpbId" : "107t982371",
"id" : "https://lobid.org/resources/107t982371",
"isbn" : [ "3-89331-303-6" ],
"inCollection" : [ {
"id" : "http://lobid.org/resources/HT013494180#!",
"type" : [ "Collection" ],
Expand Down
1 change: 1 addition & 0 deletions conf/output/test-output-32.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"type" : [ "BibliographicResource", "Article" ],
"rpbId" : "107t982372",
"id" : "https://lobid.org/resources/107t982372",
"isbn" : [ "3-89331-303-6" ],
"inCollection" : [ {
"id" : "http://lobid.org/resources/HT013494180#!",
"type" : [ "Collection" ],
Expand Down
2 changes: 1 addition & 1 deletion conf/output/test-output-37.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
"type" : [ "Contribution" ]
}, {
"agent" : {
"id" : "https://d-nb.info/gnd/Südwestrundfunk",
"id" : "Südwestrundfunk",
"label" : "Südwestrundfunk",
"type" : [ "CorporateBody" ]
},
Expand Down
3 changes: 2 additions & 1 deletion conf/output/test-output-4.json
Original file line number Diff line number Diff line change
Expand Up @@ -106,5 +106,6 @@
"fulltextOnline" : [ {
"id" : "https://nbn-resolving.org/urn:nbn:de:hbz:061:2-1643",
"label" : "urn:nbn:de:hbz:061:2-1643"
} ]
} ],
"edition" : [ "[Ausg. Ill. mit Text]" ]
}
2 changes: 1 addition & 1 deletion conf/output/test-output-47.json
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@
"type" : [ "Contribution" ]
}, {
"agent" : {
"id" : "https://d-nb.info/gnd/[Sebapharma GmbH & Co. KG, Boppard]",
"id" : "[Sebapharma GmbH & Co. KG, Boppard]",
"label" : "[Sebapharma GmbH & Co. KG, Boppard]",
"type" : [ "CorporateBody" ]
},
Expand Down
2 changes: 1 addition & 1 deletion conf/output/test-output-52.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
"type" : [ "Contribution" ]
}, {
"agent" : {
"id" : "https://d-nb.info/gnd/VCN 77",
"id" : "VCN 77",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the output for elasticsearch, right? It does not look correct that the id field is filled with a non-URI string. There is the same problem in conf/output/test-output-65.json.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's actually kind of a fix: if the value in the source field is not actually an ID (but a label), we no longer prepend the base URI. For consistency (to always have id and label), the label is used for the ID field. This might not be the final solution, but I think we can look into this when we tweak the web UI.

"label" : "VCN 77",
"type" : [ "CorporateBody" ]
},
Expand Down
4 changes: 3 additions & 1 deletion conf/output/test-output-57.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"rpbId" : "929t112198",
"id" : "https://lobid.org/resources/929t112198",
"hbzId" : "HT015954631",
"isbn" : [ "3-8031-1114-5", "978-3-8031-1114-2" ],
"inCollection" : [ {
"id" : "http://lobid.org/resources/HT013494180#!",
"type" : [ "Collection" ],
Expand Down Expand Up @@ -84,5 +85,6 @@
"label" : "Autor/in"
},
"type" : [ "Contribution" ]
} ]
} ],
"edition" : [ "11. Aufl." ]
}
2 changes: 1 addition & 1 deletion conf/output/test-output-65.json
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
"type" : [ "Contribution" ]
}, {
"agent" : {
"id" : "https://d-nb.info/gnd/SC Wassenach 1957",
"id" : "SC Wassenach 1957",
"label" : "SC Wassenach 1957",
"type" : [ "CorporateBody" ]
},
Expand Down
1 change: 1 addition & 0 deletions conf/output/test-output-68.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"type" : [ "BibliographicResource", "Book" ],
"rpbId" : "929t084686",
"id" : "https://lobid.org/resources/929t084686",
"isbn" : [ "1-904966-16-0" ],
"inCollection" : [ {
"id" : "http://lobid.org/resources/HT013494180#!",
"type" : [ "Collection" ],
Expand Down
4 changes: 3 additions & 1 deletion conf/output/test-output-81.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"type" : [ "BibliographicResource", "Book" ],
"rpbId" : "929t930040",
"id" : "https://lobid.org/resources/929t930040",
"isbn" : [ "3-920284-67-4", "3-7666-9809-5" ],
"inCollection" : [ {
"id" : "http://lobid.org/resources/HT013494180#!",
"type" : [ "Collection" ],
Expand Down Expand Up @@ -105,5 +106,6 @@
"label" : "Bearb."
},
"type" : [ "Contribution" ]
} ]
} ],
"edition" : [ "2. dt. Ausg." ]
}
1 change: 1 addition & 0 deletions conf/output/test-output-84.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"rpbId" : "107t182053",
"id" : "https://lobid.org/resources/107t182053",
"hbzId" : "HT019399122",
"isbn" : [ "978-3-506-77967-0" ],
"inCollection" : [ {
"id" : "http://lobid.org/resources/HT013494180#!",
"type" : [ "Collection" ],
Expand Down
89 changes: 89 additions & 0 deletions conf/output/test-output-85.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
{
"@context" : "http://lobid.org/resources/context.jsonld",
"type" : [ "BibliographicResource", "Book" ],
"rpbId" : "929t160199",
"id" : "https://lobid.org/resources/929t160199",
"hbzId" : "HT018859168",
"isbn" : [ "978-3-7701-8016-5" ],
"inCollection" : [ {
"id" : "http://lobid.org/resources/HT013494180#!",
"type" : [ "Collection" ],
"label" : "Rheinland-Pfälzische Bibliographie"
} ],
"title" : "Wandern auf Rheinsteig und Rheinburgenweg",
"extent" : "180 Seiten : Illustrationen, Diagramme, Karten",
"note" : [ "Enthält Index" ],
"isPartOf" : [ {
"type" : [ "IsPartOfRelation" ],
"hasSuperordinate" : [ {
"label" : "DuMont aktiv"
} ]
} ],
"responsibilityStatement" : [ "Manfred Böckling" ],
"publication" : [ {
"location" : [ "Ostfildern" ],
"startDate" : "2012",
"publishedBy" : [ "DuMont Reiseverlag" ],
"type" : [ "PublicationEvent" ]
} ],
"subject" : [ {
"id" : "http://purl.org/lobid/rpb#n102070",
"label" : "Wandern / Führer",
"type" : [ "Concept" ],
"source" : {
"id" : "http://purl.org/lobid/rpb",
"label" : "Systematik der Rheinland-Pfälzischen Bibliographie"
}
}, {
"type" : [ "ComplexSubject" ],
"label" : "Rheinsteig | Rhein-Burgen-Wanderweg | Wandern",
"componentList" : [ {
"id" : "https://d-nb.info/gnd/7501137-2",
"label" : "Rheinsteig",
"source" : {
"id" : "https://d-nb.info/gnd/7749153-1",
"label" : "Gemeinsame Normdatei (GND)"
}
}, {
"id" : "https://d-nb.info/gnd/7500776-9",
"label" : "Rhein-Burgen-Wanderweg",
"source" : {
"id" : "https://d-nb.info/gnd/7749153-1",
"label" : "Gemeinsame Normdatei (GND)"
}
}, {
"id" : "https://d-nb.info/gnd/4064532-0",
"label" : "Wandern",
"source" : {
"id" : "https://d-nb.info/gnd/7749153-1",
"label" : "Gemeinsame Normdatei (GND)"
}
} ]
} ],
"spatial" : [ {
"id" : "https://d-nb.info/gnd/4126432-0",
"label" : "Mittelrheintal",
"type" : [ "PlaceOrGeographicName" ],
"source" : {
"id" : "https://d-nb.info/gnd/7749153-1",
"label" : "Gemeinsame Normdatei (GND)"
}
} ],
"contribution" : [ {
"agent" : {
"id" : "https://d-nb.info/gnd/124591736",
"label" : "Böckling, Manfred / 1970-",
"type" : [ "Person" ]
},
"role" : {
"id" : "http://id.loc.gov/vocabulary/relators/aut",
"label" : "Autor/in"
},
"type" : [ "Contribution" ]
} ],
"description" : [ {
"id" : "http://d-nb.info/1069609889/04",
"label" : "http://d-nb.info/1069609889/04"
} ],
"edition" : [ "4., aktualisierte Auflage" ]
}
Empty file added conf/output/test-output-86.json
Empty file.
Loading
Loading