Programmatically generated Units, etc.

[mgberg]

I'm curious if anyone has given any thought about integrating with some software library to enable features like:

• Automated generation of the "mathematical" portions of QUDT (units, dimension vectors, etc.) to more easily and reliably add new units
• Easier utilization of QUDT in other software and data science projects (e.g. getting a unit object in the library by a QUDT URI or getting a QUDT URI for a unit object)

For example, Pint for Python is highly configurable and has a fairly robust representation of QUDT-related content, including but not limited to:

• Units
• Combinations of units
• Base units
• Prefixes
• Systems of units
• Compatible units
• Conversion factors (which can be precise fractions instead of floating-point approximations)
• Dimensionality
• Names and symbols
• Quantities
• Measurements (they define it as quantities with uncertainty).

A library like pint is useful to Python users because it has a lot of convenience functionality too, such as wrapping numpy and decorating functions to do conversions or enforce particular units.

However, for something like that to work reliably, it might require a new major release. For example, an extra rule may need to be added to Unit URI construction rules to enforce that the component units are always ordered alphabetically for reliable generation (i.e. unit:W-PER-K-M instead of unit:W-PER-M-K)

[steveraysteveray]

For the utilization part of your question, I'm curious if you have explored the community resources (the column of links on the right at https://qudt.org/). Do they do the kind of things you are hoping for?

PINT for the first part of your question does sound interesting, though. (I'm not a Python programmer, so it wouldn't be me to do anything with it!)

[jhodgesatmb]

Our naming convention is based, I believe, on the ordering in the dimension vector, but even if it isn’t, we would have to rearchitect the entire ontology to suit Python users, who are presumably not even using the ontology in the way it was intended (as a structured model, or they wouldn’t be using Python which is a procedural language). If the Python community is already using the QUDT vocabulary in pint then they have succeeded w/o the ontology being modified. Am I missing something here?Jack Hodges, Ph.D.Arbor StudiosOn Jun 29, 2023, at 3:00 PM, steveraysteveray ***@***.***> wrote: For the utilization part of your question, I'm curious if you have explored the community resources (the column of links on the right at https://qudt.org/). Do they do the kind of things you are hoping for? PINT for the first part of your question does sound interesting, though. (I'm not a Python programmer, so it wouldn't be me to do anything with it!) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mgberg]

@steveraysteveray: I am familiar with some of those community resources, and I've written a similar one for Python before for internal use.

@jhodgesatmb: I mentioned the capabilities Pint has for Python users just because it is an example of one of these tools that I happen to be familiar with. I'm not suggesting catering to Python users, or rearchitecting the ontology to conform to the model Pint has, or modifying Pint to conform to the QUDT model. I'm just suggesting that there could be an integration layer to enable to use QUDT and Pint together (or whatever framework and language one may choose to use, not necessarily Pint or Python).

Regardless, I think that isn't the main benefit and more of a potential side effect anyway. As @steveraysteveray mentioned, I think the more interesting part (and the reason I started this discussion) is the first part- the automatic generation of the "mathematical" portions of the definitions of Units, Prefixes, Dimension Vectors, and so on.

Currently, creating instances of these classes is a very manual process that is prone to human error, in terms of not following a convention, mathematical errors, or otherwise. For example, I recently added Stress Intensity Factor to QUDT. The process looked something like this (but generalized a bit in some places and simplified in others):

• Handle the Dimension Vector first
  • Check the dimensions of the primary unit MPa√m to verify there was no dimension vector already defined with those dimensions and create one if not (prone to human error in several ways, such as verifying the dimensional exponents are computed correctly, using the correct abbreviations for the dimensions in the right order in the URI to ensure it doesn't exist already, etc.)
  • Have a (brief) discussion about the naming convention for fractional exponents as one didn't exist in QUDT before (which would have presumably raised an exception if using an automated tool; the fix for that exception would set an automatically enforced standard for a very obscure case that other contributors or users may not know about)
  • Manually construct all the triples for the dimensional exponent values (prone to typos)
  • Manually write the LaTeX definition (which can be automated)
  • Manually construct the label (which, as it matches the lname, can be automated the same way)
• Create necessary Quantity Kind(s) (which still would need to be manually curated)
• Create all the units. For each unit:
  • Derive the lname (which requires knowledge of the convention and is prone to human error; I know the rules and I occasionally catch myself doing things like GigaPa for gigapascal or KiloG for kilogram)
  • Add the dimension vector (prone to error as making sure you select the right one with the right dimensions from the list of those labels can be a pain when a lot of dimensions are involved)
  • Manually create standardized symbols in one or more formats like Unicode or LaTeX (easy to manually create ones without knowing conventions exist, and easy to not create all the formats)
  • Manually add conversion factors (error prone, and very easy to accidently add the reciprocals of the multipliers if not careful)
  • Manually construct the label (easy to create labels that don't conform to a convention if one exists)
  • Manually add applicable unit systems (easy to automate)
  • Manually add Quantity Kinds (remains manual)
  • Manually add all annotations and identifiers (would likely remain manual)

While the Quantity Kinds and adding metadata like descriptions and informative references would still be manual and adding identifiers like IEC 61360, UNECE, and UCUM codes would probably still be manual, adding the skeletons of new units and dimension vectors could be as simple as writing out the symbol of a unit and running a script.

Furthermore, it would help ensure that the knowledge base is more complete and reliable. For example, up until I generated Unicode symbols for all the units using a common pattern, we maintained that set of symbols separately so we could guarantee a consistent UX for users. Also, there are currently about 75 units without a qudt:conversionMultiplier. There have also been a couple times I've found and fixed qudt:conversionMultipliers that were accidentally reciprocals of the correct values (e.g. 1000 instead of 0.001).

[steveraysteveray]

@mgberg, I like the direction you are suggesting, partially because I have recently been working with some colleagues (not in the context of QUDT) who did a similar thing to generate a whole set of electricity instances with a large number of permutations of number of phases, line-to-line and line-to-neutral voltages, and more. It was done using a Python program that spat out 156 results with a nice, consistent structure.

So my question is, what is the next step in your opinion? I would be happy to work with you on this as the QUDT domain person, if you would be the programmer...

[jhodgesatmb]

I wrote a web service app to do conversions a couple years ago and I began developing a similar app to do dimension checking for equations, so I agree completely about this kind of tool.I see no harm in having a layer but we are resource constrained so someone would have to step up to the plate to create and test one.Jack Hodges, Ph.D.Arbor StudiosOn Jun 30, 2023, at 7:38 AM, steveraysteveray ***@***.***> wrote: @mgberg, I like the direction you are suggesting, partially because I have recently been working with some colleagues (not in the context of QUDT) who did a similar thing to generate a whole set of electricity instances with a large number of permutations of number of phases, line-to-line and line-to-neutral voltages, and more. It was done using a Python program that spat out 156 results with a nice, consistent structure. So my question is, what is the next step in your opinion? I would be happy to work with you on this as the QUDT domain person, if you would be the programmer... —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[steveraysteveray]

@jhodgesatmb, this is not about unit conversion, it is about creating new instances of Unit, or QuantityKind, etc.

[mgberg]

@steveraysteveray Let me reach out to my organization to see if I can get permission to contribute to this.

If I can, I think next steps would include:

• Create a preliminary scope/set of requirements, e.g.:
  • What resource types we'd like to try to generate
  • What properties of each resource we'd like to try to generate
  • How to feed it what it should generate (e.g. by symbol, name, URI, etc.)
• Do a brief PoC to check the feasibility of that scope/set of requirements and see if anything else useful can be included.

[steveraysteveray]

Inspired by your suggestion, and with the help of ChatGPT since I'm not really a programmer, I created the following POC in Python for prefixed units of the meter. (A good Friday afternoon distraction). Is this the kind of thing you are thinking of?

import decimal
from rdflib import Graph, Literal, Namespace, BNode, URIRef
from rdflib.namespace import RDF, RDFS, XSD, DC

# define namespaces
qudt = Namespace("http://qudt.org/schema/qudt/")
sou = Namespace("http://qudt.org/vocab/sou/")
quantitykind = Namespace("http://qudt.org/vocab/quantitykind/")
prefix_ns = Namespace("http://qudt.org/vocab/prefix/")
unit = Namespace("http://qudt.org/vocab/unit/")
quantitykind = Namespace("http://qudt.org/vocab/quantitykind/")

# initialize graph
g = Graph()
# Bind the qudt and dcterms prefixes to their respective URIs
g.bind("qudt", qudt)
g.bind("dcterms", DC)
g.bind("sou", sou)
g.bind("quantitykind", quantitykind)

# define SI prefixes
si_prefixes = {
    'Yotta': 1e24, 'Zetta': 1e21, 'Exa': 1e18, 'Peta': 1e15, 'Tera': 1e12,
    'Giga': 1e9, 'Mega': 1e6, 'Kilo': 1e3, 'Hecto': 1e2, 'Deca': 10,
    'Deci': 1e-1, 'Centi': 1e-2, 'Milli': 1e-3, 'Micro': 1e-6, 'Nano': 1e-9,
    'Pico': 1e-12, 'Femto': 1e-15, 'Atto': 1e-18, 'Zepto': 1e-21, 'Yocto': 1e-24
}

def format_float(f):
    d1 = decimal.Decimal(repr(f))
    return format(d1, 'f')

# populate graph
for prefix, multiplier in si_prefixes.items():
    prefix_node = unit[prefix + "M"]  # use "M" instead of "Meter"
    g.add((prefix_node, RDF.type, qudt.DerivedUnit))
    g.add((prefix_node, RDF.type, qudt.Unit))
    g.add((prefix_node, DC.description, Literal(f"One {prefix.lower()}meter equals exactly {multiplier} meters.")))
    g.add((prefix_node, qudt.applicableSystem, sou.CGS))
    g.add((prefix_node, qudt.applicableSystem, sou.CGS_EMU))
    g.add((prefix_node, qudt.applicableSystem, sou.CGS_GAUSS))
    g.add((prefix_node, qudt.applicableSystem, sou.SI))
# convert to non-scientific format with desired precision
    multiplier_str = format_float(multiplier)
    g.add((prefix_node, qudt.conversionMultiplier, Literal(multiplier_str, datatype=XSD.string)))
    g.add((prefix_node, qudt.dbpediaMatch, Literal(f"http://dbpedia.org/resource/{prefix}metre", datatype=XSD.anyURI)))
    g.add((prefix_node, qudt.hasDimensionVector, qudt.A0E0L1I0M0H0T0D0))
    g.add((prefix_node, qudt.hasQuantityKind, quantitykind.Length))
    g.add((prefix_node, qudt.informativeReference, Literal(f"http://en.wikipedia.org/wiki/{prefix}metre", datatype=XSD.anyURI)))
    g.add((prefix_node, qudt.prefix, prefix_ns[prefix]))
    g.add((prefix_node, qudt.symbol, Literal(f"Needs symbol for {prefix}meter", lang="en")))
    g.add((prefix_node, qudt.ucumCode, Literal(f"Needs UCUM code", datatype=qudt.UCUMcs)))
    g.add((prefix_node, RDFS.isDefinedBy, Literal("http://qudt.org/2.1/vocab/unit", datatype=XSD.anyURI)))
    g.add((prefix_node, RDFS.label, Literal(f"{prefix}metre", lang="en")))

# First, serialize the graph
rdf_string = g.serialize(format='turtle')
# Then, replace the unwanted parts
rdf_string = rdf_string.replace('conversionMultiplier "', 'conversionMultiplier ')
rdf_string = rdf_string.replace('"^^xsd:string', '')
# Finally, write the modified string to the file
with open("Meter.ttl", "w") as file:
    file.write(rdf_string)

[mgberg]

That's fun, I've tried to get ChatGPT to use QUDT too.

That's a simplified version of what we might be able to accomplish using a library like Pint. That example simply creates a graph using explicit information (e.g. the dimension vector and quantity kind is explicitly provided) and is probably fragile in places (like the construction of DBPedia/Wikipedia links wouldn't work like that for everything).

Using Pint as an example:

• It dimensionally aware, so we could automatically generate dimension vectors and easily connect units to them.
• It understands prefixes, so we could automatically generate prefixes and perhaps do other things like automatically add qudt:prefix and qudt:isScalingOf (although qudt:isScalingOf was deprecated, so I suppose not that- why was that deprecated?)
• It understands systems of units, so those can all be autogenerated and connected
• Conversion factors are derived based on the composition of units
• It can parse/create textual representations of units (like "meter / second^2" or "m/s²" or "meter per second squared"), and the way it does so is configurable and customizable
• Using a translation library, it can generate labels in whatever languages we would be interested in

If I get approval to do so, I could just play around with it and see what I can come up with as well.

[steveraysteveray]

Cool.
To answer your question about "isScalingOf", we deprecated because it didn't generalize well with multiple systems of units. It would have required multiple entries, one per system. In the end, we decided this information is derivable anyway.

[mgberg]

Ah, makes sense- I didn't think about multiple systems.

I'll let you know if I can investigate this.

[jimkont]

Hi, related to this thread, I am also interested in understanding complex units such as

• (L/day)/(ug/kg) (Liter per Day per Microgram per Kilogram),
• (mL/min)/(mg/kg) (Milliliter per Minute per Milligram per Kilogram),
• d.g/ml/kg (Day Times Gram Per Milliliter Per Kilogram),
• ml/(mg/kg/d) (Milliliter per Milligram per Kilogram per Day),
• min.umol/L/g. (Minute Times Micromole Per Liter Per Gram) etc,

there is alist of ucum-based codes like the ones above that are used in lab-related results that are relevant and missing from QUDT. There are different directions one could take, listing some possible ones in case they have already been discussed in the past.

Manually add these units to QUDT. This would be time-consuming, but get more peer-review of the added units.

If a unit is not defined explicitly in QUDT, can we define it as a composite unit using a unit formula?
e.g. (L/day)/(ug/kg) could be defined as formula between unit:L-PER-DAY divided by unit:MicroGM-PER-KiloGM
in this case, do we have all the needed metadata to create i.e. unit:L-PER-DAY-PER-MicroGM-PER-KiloGM (dummy IRI here) as a concrete QUDT unit?
If some critical metadata are missing, can we create some abstract unit composition type that can only facilitate conversions using the conversion factors from the unit components, e.g. (dL/hour)/(mg/Kg), (mL/min)/(mg/kg), etc

I understand that unit composition information is also stored in the dimensions vector, but is this sufficient to identify the core units one unit is composed of? e.g.

• unit:L-PER-DAY is Volume (L^3) / Time (T^-1)
• unit:MilliGM-PER-M3-HR is Mass (M1), volume (L^-3) and Time (T^-1)
  This works in simple cases, but does this generalize in more complex units?
  This information could be used to decompose existing QUDT units and enable conversions to more targets that are not explicitly defined in QUDT, i.e. Decilitre Per Day (possible target unit)

Any thoughts or pointers to older discussions on the topics would be very appreciated (cc/ @glow-mdsol)

[steveraysteveray]

Hi @jimkont, I have a few thoughts on this:

1. We do have the relations qkdvNumerator and qkdvDenominator that are sometimes used (in addition to the hasDimensionVector relation) to distinguish dimensionless QuantityKinds as ratios of two equally dimensioned values (documented here). To date, we have only used that structure for dimensionless QuantityKinds. It would be awkward to use that approach for units unless we were sure that was the only interpretation - it does not generalize. But one can navigate from the Unit to the (possibly multiple) QuantityKinds to find such structures, as illustrated in the documentation referenced above.
2. Because we only allow one -PER- in our URIs to avoid ambiguity, your example (L/day)/(ug/kg) would become unit:L-KiloGM-PER-DAY-MicroGM. More importantly, the final dimension vector would be qkdv:A0E0L3I0M0H0T-1D0, and 5 QuantityKinds have that dimensionality:

• MoistureDiffusivity
• RecombinationCoefficient
• SoundVolumeVelocity
• VolumeFlowRate
• VolumePerUnitTime

An interesting question is whether any of those QuantityKinds are appropriate to use the qkdvNumerator and qkdvDenominator relations. Clearly VolumeFlowRate does not carry that meaning, but I'd have to investigate whether MoistureDiffusivity has that underlying dimensionality. If not, would it make sense to define a new QuantityKind that does? You tell me - I'm not expert in your use case.

All this speaks to the question of how much could be automatically generated. I would suggest that it's the QuantityKind that distinguishes concepts that might share the same DimensionVector, so perhaps the focus should be less on the Unit and more on the QuantityKind. Put another way, just because two Units have the same dimensionality does not mean they are commensurate, which is why we added the qkdvNumerator and qkdvDenominator relations as a partial fix. You can't really compare a relative humidity (dimensionless) with an interest rate (also dimensionless), which seems obvious when stated this way (as QuantityKinds), but not obvious when comparing two percentages (unit:PERCENT).

So circling back to your original question, can you provide what the QuantityKinds are for the Units you are proposing? How are these Units used?

And finally, because this gets back to context (for which the QuantityKind is a hint), it's hard for me to see how one could generate such Units and QuantityKinds at scale. I'm afraid it's still a manual, peer-reviewed process. What we need is an army of Wikipedia-like editors! Want to volunteer?

[dr-shorthair]

Yes, QuantityKind is the key. But it carries semantics which is not so easily automated.

FWIW BIPM have a prototype service called the 'SI Reference Point' which is doing something similar to QUDT, but only for the SI units (just 29 so far).
It is password protected at present, but uses URIs like https://si-digital-framework.org/si-unit/A
They have a list of 29 Quantities (which QUDT calls QuantityKinds) one per Unit from their system

electric current
The kind of quantity electric current has the unit ampere (symbol A). (code: ELCU)
luminous intensity
The kind of quantity luminous intensity has the unit candela (symbol cd). (code: LUIN)
thermodynamic temperature
The kind of quantity thermodynamic temperature has the unit kelvin (symbol K). (code: TEMT)
mount of substance
The kind of quantity amount of substance has the unit mole (symbol mol). (code: AMSU)
mass
The kind of quantity mass has the unit kilogram (symbol kg). (code: MASS)
length
The kind of quantity length has the unit metre (symbol m). (code: LENG)
time
The kind of quantity time has the unit second (symbol s). (code: TIME)
activity referred to a radionuclide
The kind of quantity activity referred to a radionuclide has the unit becquerel (symbol Bq). (code: ARRN)
electric charge
The kind of quantity electric charge has the unit coulomb (symbol C). (code: ELCH)
Celsius temperature
The kind of quantity Celsius temperature has the unit degree Celsius (symbol °C). (code: TEMC)
capacitance
The kind of quantity capacitance has the unit farad (symbol F). (code: ELCA)
absorbed dose
The kind of quantity absorbed dose has the unit gray (symbol Gy). (code: ABDO)
inductance
The kind of quantity inductance has the unit henry (symbol H). (code: ELIN)
frequency
The kind of quantity frequency has the unit hertz (symbol Hz). (code: FREQ)
catalytic activity
The kind of quantity catalytic activity has the unit katal (symbol kat). (code: CATA)
luminous flux
The kind of quantity luminous flux has the unit lumen (symbol lm). (code: LUFL)
illuminance
The kind of quantity illuminance has the unit lux (symbol lx). (code: ILLU)
force
The kind of quantity force has the unit newton (symbol N). (code: FORC)
electric resistance
The kind of quantity electric resistance has the unit ohm (symbol
). (code: ELRE)
pressure
The kind of quantity pressure has the unit pascal (symbol Pa). (code: PRES)
electric conductance
The kind of quantity electric conductance has the unit siemens (symbol S). (code: ELCO)
dose equivalent
The kind of quantity dose equivalent has the unit sievert (symbol Sv). (code: DOEQ)
solid angle
The kind of quantity solid angle has the unit steradian (symbol sr). (code: ANGS)
magnetic flux density
The kind of quantity magnetic flux density has the unit tesla (symbol T). (code: MGFD)
electric potential difference
The kind of quantity electric potential difference has the unit volt (symbol V). (code: ELPD)
power
The kind of quantity power has the unit watt (symbol W). (code: POWR)
magnetic flux
The kind of quantity magnetic flux has the unit weber (symbol Wb). (code: MGFL)
energy
The kind of quantity energy has the unit joule (symbol J). (code: ENGY)
plane angle
The kind of quantity plane angle has the unit radian (symbol rad). (code: ANGP)

[jimkont]

thanks @steveraysteveray and @dr-shorthair, for the replies and your valuable insights! I see that, indeed, quantitykinds is the key here. We can automate up to one point, but a manual process will be required in the end.
I also understand now that the unit formula I was referring to, should actually be a quantityKind formula, which is a bit harder to automate since each unit can have multiple quantity kinds. We could possibly go only with standard units but this needs some more thought and experimentation. I'll take this back to the team for discussion.

I would be happy to contribute my findings to the community, but I will be working on the engineering side of this problem. I am not a domain expert for such units, but we will probably be able to find some.

[fkleedorfer]

@mgberg You might want to look at our qudtlib project. It produces artifacts for java (qudtlib-java) and javascript (qudtlib-js) that contain a domain model based on QUDT and unit/quantitykind/prefix/systemOfUnits instances - without any RDF processing at runtime. Rather, java/js code is generated at compile time and packaged in the artifacts. Maybe a python variant (based on Pint) would address your needs? I could help setting that up.

[mgberg]

Based on my understanding of those projects (which is admittedly limited so I may be wrong), these projects supply (as you mention) a domain model and instances that are automatically generated from the RDF representation. As I mentioned in the original post, I think a Python version of that is a good idea. I wouldn't mind contributing to that, and I would definitely use it. I've written a version of it for internal use, but it builds the instances dynamically at runtime and caches them.

What I'm primarily talking about in this discussion is something that automatically generates the RDF instance data such that the "math" portions can be done programmatically. This hopefully would make adding new units/quantity kinds/dimension vectors/etc. easier, help standardize the way in which multiplication factors and offsets are computed (e.g. number of decimal places/significant figures), and in general would additions more reliable. It would also have the side effect of making the RDF data fed into the qudtlib projects more reliable. As mentioned above, much of the metadata would likely still be manual.

I think (again based on my limited understanding of those projects) that the programmatically generated data capability is different than what qudtlib currently provides, but I definitely think both are useful and important.

On a related note, I should have some time to play around with this in the near future myself.

[fkleedorfer]

I'm working on a contribution tool, which is a framework to be used in a Java class (just its main method) to prepare a contribution to QUDT. I found myself having to query the model repeatedly to see what's already there (e.g. quantityKind/unit by dimension vector, by name, etc.) so there is limited support for that, and there is some support for creating new units and quantitykinds.

Maybe you want to have a look and see if it would fit your needs. I hope to release this fairly soon (maybe in 2 weeks)

[fkleedorfer]

oh. forgot the link