Make dates available for SPARQL #3

mzeinstra · 2021-02-11T14:21:28Z

I don't see this as an open ticket yet.

We discussed that the MVP would be to expose the lowest date of an EDTF value to SPARQL in Wikibase. Given possibilities this could be the highest and lowest values.

This is to make the current operators on dates available in SPARQL.

JeroenDeDauw · 2021-02-12T15:34:41Z

2021-02-12 status: high priority

JeroenDeDauw · 2021-03-14T00:19:29Z

@mzeinstra what to do for seasons? Start month? End month? Middle month? Both start and end month?

mzeinstra · 2021-03-14T07:01:30Z

Just out of the top of my head. Wouldn’t last day and first day of the season not work?

…

On Sun, 14 Mar 2021 at 01:19, Jeroen De Dauw ***@***.***> wrote: @mzeinstra <https://github.com/mzeinstra> what to do for seasons? Start month? End month? Middle month? Both start and end month? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK54J7DWKLBOVD47XHHOJTTDP6J7ANCNFSM4XO5KB2Q> .

JeroenDeDauw · 2021-03-14T14:47:52Z

Implementation wise we can make it all happen. Usecase wise the first and last day for seasons works for some cases but not for others. Some examples where it does not work:

Showing a timeline of when people where born (in case of a season they would show twice)
Finding all conferences that happened more than once in 2020 (Spring 2020 shows up twice)

I also suspect that using start and end month is better than using the days. The dates also come with precision, and month precision is closer to season than day precision.

If we go with multiple values (ie start and end month), then perhaps it might even make sense to include all months part of the season?

JeroenDeDauw · 2021-03-14T14:55:39Z

Another example of a use case that gets messed up, this time applicable to intervals, when using start and end time:

Imagine having interval 1900-1999 (20th century). If you do a query finding times between 1890 and 1910, you will find the item via the starting time. But if you query for all times between 1910 and 1930, you will not find the item at all. It is not clear to me how to solve that, and it might not be possible without adding features to Blazegraph. And I can imagine, that for the people running the queries, it might be best if intervals (and possibly seasons and sets) are skipped. It is possible we make queries less usable for them by including these. Hard to tell.

mzeinstra · 2021-03-14T17:43:05Z

I agree on adding the months in a season, that seems to be the best way forward.

For intervals we could add all years in a sequence, but that might not be the best solution in this. Your proposal is to not expose intervals at all?

I assume we will also expose the 'raw' EDTF string as well?

JeroenDeDauw · 2021-03-14T18:31:58Z

Your proposal is to not expose intervals at all?

Yes. That could be added without all the guessing once a concrete usecase materializes, which might well be never.

I assume we will also expose the 'raw' EDTF string as well?

At the moment not. The EDTF is being translated into standard Wikibase time values, so we get maximum compatibility with tools. I am not sure what the implications of also exposing it in RDF as a string are, and am worried that having some values for P123 be a date and some be a string is a big no-no. So this would need some investigation, which will either take a lot longer than the technical work itself, or uncover the need for a bunch of extra technical work.

mzeinstra · 2021-03-14T21:03:01Z

I agree, that is a good start within the limited time.

I am afraid that not exposing the EDTS-as-string might also close the route of exporting EDTF data from the platform. Would that be true, or would it only be for the Sparql?

JeroenDeDauw · 2021-03-15T00:28:08Z

I was just talking about SPARQL and RDF. The standard MediaWiki and Wikibase export mechanisms contain the string version. Example of entity JSON with some EDTF strings in it via the web API: http://edtf.wikibase.wiki/w/api.php?action=wbgetentities&ids=P1

So not having the string in SPARQL or RDF does not create an export issue. Indeed, the RDF does not contain the entirety of the native Wikibase time values.

mzeinstra · 2021-03-15T08:10:30Z

Ah ok, than that use case doesn't exists anymore thanks.

JeroenDeDauw · 2021-03-18T17:24:08Z

Implementation should be done. Now we will test if the SPARQL queries actually work.

Example time-based query on Wikidata: https://query.wikidata.org/#SELECT%20%3Fitem%20%3FitemLabel%20%3Fdate_of_birth%20WHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP31%20wd%3AQ146.%0A%20%20%20%20%20%20%20%20%3Fitem%20wdt%3AP569%20%3Fdate_of_birth.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%0A%20%20FILTER%28%28%3Fdate_of_birth%20%3E%20%222015-01-01%22%5E%5Exsd%3AdateTime%29%20%26%26%20%28%3Fdate_of_birth%20%3C%20%222021-06-01%22%5E%5Exsd%3AdateTime%29%29%0A%7D%0AORDER%20BY%20DESC%28%3Fdate_of_birth%29

SELECT ?item ?itemLabel ?date_of_birth WHERE {
  ?item wdt:P31 wd:Q146.
        ?item wdt:P569 ?date_of_birth.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
  FILTER((?date_of_birth > "2015-01-01"^^xsd:dateTime) && (?date_of_birth < "2021-06-01"^^xsd:dateTime))
}
ORDER BY DESC(?date_of_birth)

JeroenDeDauw · 2021-03-18T18:15:14Z

Working as expected \o/

JeroenDeDauw · 2021-03-18T18:17:17Z

I think we can close this task and open more specific tickets if some further tweaks are needed.

mzeinstra · 2021-03-19T08:12:37Z

Agreed, we will test this and get back to you. with specific tickets.

mzeinstra · 2021-03-19T08:54:44Z

@JeroenDeDauw can we use the environment that you set up to test this? http://edtf.wikibase.wiki/wiki/Property:P1 I don't see the query service available there.

JeroenDeDauw · 2021-03-19T12:32:30Z

I will update the demo instance later today so you can test queries there later today, and ping you once it happened.

JeroenDeDauw · 2021-03-19T18:17:53Z

Query service now available on demo instance: http://edtf.wikibase.wiki:8282/#SELECT%20%3Fitem%20%3FitemLabel%20%3Fedtf_date%20WHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP1%20%3Fedtf_date.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%0A%7D%0AORDER%20BY%20DESC%28%3Fedtf_date%29

andrawaag · 2021-03-23T11:57:57Z

I was pointed to this by @mzeinstra. Not having the EDTF representation in Blazegraph might be an issue for downstream pipelines that ingest or export to and from CIDOC-CRM represented data values. CIDOC-CRM is RDF. I am wondering if this issue could be resolved if the dates in EDTF are already in Wikibase native transformed in the native date stamp of Wikibase, where the EDTF representation is maintained as a qualifier to those statements. This would look something like this:

https://safsandbox1.wiki.opencura.com/wiki/Item:Q1
(This example is on wbstack which does not got the EDTF extension, I have selected string as data type.)

Wouldn't using qualifier maintain the integrity with EDTF also in the RDF representation?

mzeinstra · 2021-03-23T16:27:41Z

@andrawaag I thought you suggested creating a 'hidden' qualifier to contain the string and not the other way around. right?

andrawaag · 2021-03-24T08:49:18Z

No, I would not hide this. My point is that conceptually Wikibase consists of two redudent data layers, a relation model and a RDF model. We should not remove this redundancy. The RDF layer is crucial for information retrieval since querying wikibase through the API is suboptimal. It is not possible to query Wikibase on both strings and statements. Here the WBQS is key. If there is a discrepancy between the two models, information retrieval will become difficult.

THere is indeed a difficulty in EDTF is one would like to do sorting, especially if the model captures only the string representation. If the sollution is transforming the EDTF time string to a XSD:Datetime value I would do that in both layers, and the suggestion I made is one possibility.

But I would not hide that, on the contrary, that would lead to downstream confusion,

mzeinstra · 2021-03-24T16:19:04Z

@JeroenDeDauw I'll have a further discussion with @andrawaag on this.

In the meantime. I was wondering if it is possible to something like this with Blazegraph:

PersonX birthDate “~2021-XX-05?“^^xsd:string .
PersonX birthDate “2021”^^xsd:datetime .

That way we could move the responsibility of searching through the dates to the person creating the query.

JeroenDeDauw · 2021-03-24T16:43:04Z

I am not familiar with RDF or Blazegraph, so can't tell what is appropriate or what will work without prior investigation.

What I do know is what Wikibase outputs as RDF for dates:

<rdf:Description rdf:about="http://edtf.wikibase.wiki/value/103522c70e98676f031e23d4ed0c5ea6">
  <rdf:type rdf:resource="http://wikiba.se/ontology#TimeValue"/>
  <wikibase:timeValue rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2006-05-01T00:00:00Z</wikibase:timeValue>
  <wikibase:timePrecision rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">10</wikibase:timePrecision>
  <wikibase:timeTimezone rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">0</wikibase:timeTimezone>
  <wikibase:timeCalendarModel rdf:resource="http://www.wikidata.org/entity/Q1985727"/>
</rdf:Description>

See the bottom of: view-source:http://edtf.wikibase.wiki/wiki/Special:EntityData/Q1.rdf

This includes

ISO timestamp
precision
time zone
calendar model

It does not include before or after, both part of standard Wikibase time values. So you cannot get access to the full data via RDF or SPARQL in stock Wikibase.

I suspect we can add more fields to the above RDF Description without breaking the query service. So we could add the EDTF string as such. I am not sure it is "correct" to do this from an RDF perspective. And I am unsure to which degree the information will be queryable via SPARQL.

So to change things here, I either need a specification of what the desired RDF output is, or I first need to investigate these topics more so I can make an informed recommendation.

mzeinstra · 2021-03-26T13:43:51Z

I had a discussion with Andra on this functionality.

To be able to have the proper functionality for export in RDF and for presentation in SPARQL we that you StringValue aftere you add the TimeValues here:

WikibaseEdtf/src/Services/RdfBuilder.php

Line 31 in 1d9f772

    
           public function addValue( RdfWriter $writer, $propertyValueNamespace, $propertyValueLName, $dataType, $snakNamespace, PropertyValueSnak $snak ) {

As you say it will most likely not break BlazeGraph and it will help us to present the EDTF string in SPARQL as well use the TTL and RDF export possibility in e.g. http://edtf.wikibase.wiki/wiki/Special:EntityData/Q2.ttl

Would that work @JeroenDeDauw ?

After this export I will ask @andrawaag and Jose Labra to verify if that is working as expected.

JeroenDeDauw · 2021-03-26T14:12:23Z

To be able to have the proper functionality for export in RDF and for presentation in SPARQL we that you StringValue aftere you add the TimeValues here

huh?

mzeinstra · 2021-03-26T14:24:54Z

Do you want to have a call on this today? e.g. at 16:00?

JeroenDeDauw · 2021-03-26T14:43:36Z

I send you an invite

This is for #3 Note: I am unsure if this output makes sense RDF/SPARQL wise.

JeroenDeDauw · 2021-03-28T16:09:22Z

The RDF now also contains the plain EDTF as a string.

The above item results in: https://pastebin.com/gGiRPGAb. (Search for [2022] to find the plain EDTF)

(Not deployed on demo system yet)

mzeinstra · 2021-04-02T07:04:13Z

I've asked Andra and Jose if this works for their use cases as well. Could you make this available on the demo system? So we can test Sparql as well.

JeroenDeDauw · 2021-04-04T17:54:49Z

Could you make this available on the demo system?

Done

mzeinstra · 2021-04-06T11:27:22Z

Interesting.

I see that it appears in the ttl files e.g. (http://edtf.wikibase.wiki/wiki/Special:EntityData/P1.ttl)

wdt:P1 "[2020, 2021, 2022, 2023, 2024]"^^xsd:edtf,
		"2020-01-01T00:00:00Z"^^xsd:dateTime,
		"2021-01-01T00:00:00Z"^^xsd:dateTime,
		"2022-01-01T00:00:00Z"^^xsd:dateTime,
		"2023-01-01T00:00:00Z"^^xsd:dateTime,
		"2024-01-01T00:00:00Z"^^xsd:dateTime,

But then I expect the following to works too, right? @andrawaag

SELECT ?item ?itemLabel ?edtf_date WHERE {
  ?item wdt:P1 "[2020, 2021, 2022, 2023, 2024]"^^xsd:edtf.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}
ORDER BY ASC(?edtf_date)

andrawaag · 2021-04-08T10:14:27Z

Jose and I reviewed the TTL file at http://edtf.wikibase.wiki/wiki/Special:EntityData/Q1.ttl. By adding the EDTF type as a string to the RDF representation allows the roundtripping which is crucial to maintain the data integrity inside Wikibase.

We do have some concerns though regarding the transformation from edtf to xsd:datetime. In the current implementation, edtf is stated as xsd:edtf, which is incorrect. EDTF is not part of XSD. Can this be changed to the applicable namespace? e.g. (https://id.loc.gov/datatypes/edtf.html)

Would it be possible to document the rules that are used to transform between edtf and xsd:datetime. For example in the above cite rdf representation of Q1 we see:
wdt:P1 "2006-24"^^xsd:edtf, "2006-12-01T00:00:00Z"^^xsd:dateTime, "2006-01-01T00:00:00Z"^^xsd:dateTime, "2006-02-01T00:00:00Z"^^xsd:dateTime,

2006-24 is to represent winter of 2006. This seems to be transformed to January, February and December 2006. Is that correct, because one could argue that it actually is 2005-12, 2006-1, 2006-2 or 2006-12, 2007-1, 2007-2. If you are from e.g. Australia or Chile, that might be 2006-7, 2006-8, 2006-9.

JeroenDeDauw · 2021-04-08T17:06:25Z

We do have some concerns though regarding the transformation from edtf to xsd:datetime. In the current implementation, edtf is stated as xsd:edtf, which is incorrect. EDTF is not part of XSD. Can this be changed to the applicable namespace? e.g. (https://id.loc.gov/datatypes/edtf.html)

#13

GreenReaper · 2021-04-27T06:39:36Z

Your proposal is to not expose intervals at all?

Yes. That could be added without all the guessing once a concrete usecase materializes, which might well be never.

It is true that for some it could be problematic, but for the day within a month or month within a year cases it's useful to have at least one date. Here's my use case:

I want a database of multi-day events and instances of those so I can show them on maps, lists, etc. I might also want to transfer this data into metadata for other consumers, e.g. JSON-LD Event.

In some of these cases, I want to retrieve the most recent or upcoming instance of a annual event, that would be determined via a SPARQL query.

Previously I'd intended to use start time statements with end time qualifiers to allow for the possibility of cancellation or rescheduling, which I also want to record and (in some cases) show.

The new type might be better for this, because the period/interval itself is the subject of a single claim.

However, without a datetime value in SPARQL I'd likely have to do my own sorting through items and parsing of dates (or, possibly, pass in a lot of matching text-mode filters) to get the right one.

This could include uncertain dates, e.g. "Eurofurence (likely August 2022)". I'd normally consider this to fall on the start of that month, or - less preferred, but maybe nice to have as well - the end of the month.

Presumably it'd use day precision per the example above, or less for month or year intervals. (Failing that, given the distribution of time zones, it might be best to use ~11:00 UTC on the day rather than midnight, to avoid being in a different day in some locations.)

JeroenDeDauw added the enhancement New feature or request label Feb 11, 2021

JeroenDeDauw self-assigned this Feb 12, 2021

JeroenDeDauw mentioned this issue Mar 15, 2021

Do not turn intervals into two dates in RDF #8

Merged

JeroenDeDauw added a commit that referenced this issue Mar 28, 2021

Add plain EDTF value to RDF

bb9e9b8

This is for #3 Note: I am unsure if this output makes sense RDF/SPARQL wise.

GreenReaper mentioned this issue Apr 27, 2021

Should Intervals and Sets support dates with times? ProfessionalWiki/EDTF#70

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make dates available for SPARQL #3

Make dates available for SPARQL #3

mzeinstra commented Feb 11, 2021

JeroenDeDauw commented Feb 12, 2021

JeroenDeDauw commented Mar 14, 2021

mzeinstra commented Mar 14, 2021 via email

JeroenDeDauw commented Mar 14, 2021 •

edited

Loading

JeroenDeDauw commented Mar 14, 2021

mzeinstra commented Mar 14, 2021

JeroenDeDauw commented Mar 14, 2021

mzeinstra commented Mar 14, 2021

JeroenDeDauw commented Mar 15, 2021 •

edited

Loading

mzeinstra commented Mar 15, 2021

JeroenDeDauw commented Mar 18, 2021 •

edited

Loading

JeroenDeDauw commented Mar 18, 2021

JeroenDeDauw commented Mar 18, 2021

mzeinstra commented Mar 19, 2021

mzeinstra commented Mar 19, 2021

JeroenDeDauw commented Mar 19, 2021

JeroenDeDauw commented Mar 19, 2021

andrawaag commented Mar 23, 2021

mzeinstra commented Mar 23, 2021

andrawaag commented Mar 24, 2021

mzeinstra commented Mar 24, 2021

JeroenDeDauw commented Mar 24, 2021

mzeinstra commented Mar 26, 2021

JeroenDeDauw commented Mar 26, 2021

mzeinstra commented Mar 26, 2021

JeroenDeDauw commented Mar 26, 2021

JeroenDeDauw commented Mar 28, 2021

mzeinstra commented Apr 2, 2021

JeroenDeDauw commented Apr 4, 2021

mzeinstra commented Apr 6, 2021

andrawaag commented Apr 8, 2021 •

edited

Loading

JeroenDeDauw commented Apr 8, 2021

GreenReaper commented Apr 27, 2021 •

edited

Loading

Make dates available for SPARQL #3

Make dates available for SPARQL #3

Comments

mzeinstra commented Feb 11, 2021

JeroenDeDauw commented Feb 12, 2021

JeroenDeDauw commented Mar 14, 2021

mzeinstra commented Mar 14, 2021 via email

JeroenDeDauw commented Mar 14, 2021 • edited Loading

JeroenDeDauw commented Mar 14, 2021

mzeinstra commented Mar 14, 2021

JeroenDeDauw commented Mar 14, 2021

mzeinstra commented Mar 14, 2021

JeroenDeDauw commented Mar 15, 2021 • edited Loading

mzeinstra commented Mar 15, 2021

JeroenDeDauw commented Mar 18, 2021 • edited Loading

JeroenDeDauw commented Mar 18, 2021

JeroenDeDauw commented Mar 18, 2021

mzeinstra commented Mar 19, 2021

mzeinstra commented Mar 19, 2021

JeroenDeDauw commented Mar 19, 2021

JeroenDeDauw commented Mar 19, 2021

andrawaag commented Mar 23, 2021

mzeinstra commented Mar 23, 2021

andrawaag commented Mar 24, 2021

mzeinstra commented Mar 24, 2021

JeroenDeDauw commented Mar 24, 2021

mzeinstra commented Mar 26, 2021

JeroenDeDauw commented Mar 26, 2021

mzeinstra commented Mar 26, 2021

JeroenDeDauw commented Mar 26, 2021

JeroenDeDauw commented Mar 28, 2021

mzeinstra commented Apr 2, 2021

JeroenDeDauw commented Apr 4, 2021

mzeinstra commented Apr 6, 2021

andrawaag commented Apr 8, 2021 • edited Loading

JeroenDeDauw commented Apr 8, 2021

GreenReaper commented Apr 27, 2021 • edited Loading

JeroenDeDauw commented Mar 14, 2021 •

edited

Loading

JeroenDeDauw commented Mar 15, 2021 •

edited

Loading

JeroenDeDauw commented Mar 18, 2021 •

edited

Loading

andrawaag commented Apr 8, 2021 •

edited

Loading

GreenReaper commented Apr 27, 2021 •

edited

Loading