Add KHR_interactivity draft

[lexaknyazev]

Rendered version: https://github.com/KhronosGroup/glTF/blob/interactivity/extensions/2.0/Khronos/KHR_interactivity/Specification.adoc

[bhouston]

Great work! What sections are next? Also what could we release as a draft extension for SIGGRAPH 2023?

[aaronfranke]

If we have Pi, we should also have Tau. https://tauday.com/tau-manifesto The value Tau is the circle constant (equal to 2*Pi, or rather, Pi is half of Tau). Using Tau usually results in more readable code. Tau is supported in many programming languages such as C#, Java, Python, GDScript, Rust, Unreal Blueprints, and more, so it's useful for interoperability with other languages, especially Unreal Blueprints which is conceptually similar.

===== Tau

[cols="1h,1,2"]
|===
| Type | `math/tau` | The circle constant, the circumference of the unit circle in radians.
| Output value sockets | `float value` | 6.2831853071795862
|===

[aaronfranke]

In case more justification is needed, the KHR_interactivity document currently contains Degrees-To-Radians (math/rad) and Radians-To-Degrees (math/deg). These allow converting between degrees and radians. Tau can be used for a similarly important purpose, dividing or multiplying by this number allows converting between turns and radians, where 1 turn is τ radians. Tau can be used for more than just this, but the point is, if math/rad and math/deg are justified as a part of KHR_interactivity for converting angles, then math/tau is also justified for converting angles.

[aaronfranke]

Why are variables accessed by index instead of by name? It would make more sense to me to set a variable called "money" than one called 27.

[Hackn0214]

When defining a custom event as shown below, if event/receive and event/send nodes refer to this custom event.

 "events": [
    {
      "id": "MyEvent",
      "values": [
        {
          "id": "Val_1",
          "type": 0
        },
        {
          "id": "Val_2",
          "type": 0
        }
      ]
    }
  ],

I think the generated node will have sockets as shown below, but what do you think?

[Hackn0214]

Is it possible to return the value pointed to by a pointer with the behavior graph?
For example, I would like to change the base color of a selected object's material.

/nodes/{nodeIdx}/mesh -> meshIdx
/meshes/{meshIdx}/primitibes/0/material -> materialIdx
/materials/{materialIdx}/pbrMetallicRoughness/baseColorFactor

[boguscoder]

If configuration value is array of length 2,3,4 or 16 it could be ambiguous if its vec2/vec3/vec4/vec4x4 or int[], this might not be a problem for JS but is a problem for strongly-typed implementations, is there any recommendation for them how to disambiguate?

[Hackn0214]

For example, how about specifying the type at the time of the call?
In fact, I would also like to have a feature that returns the number of elements as shown below.
/materials -> total material number in the scecne
/meshes/0/primitives -> Total primitive number

[boguscoder]

@Hackn0214 you already can get number of materials when using appropriate pointer , check out core pointers

[Hackn0214]

That's wonderful!
Thank you!

[Hackn0214]

I would appreciate it if you could add /meshes/{}/primitives.length

[javagl]

Is there a (reasonably up-to-date) machine-processable representation of this? (I.e. something like a schema or a "repository" of node structure descriptions)?

[lexaknyazev]

Is there a (reasonably up-to-date) machine-processable representation of this? (I.e. something like a schema or a "repository" of node structure descriptions)?

Not yet, this document is the only normative source for now. We'll publish regular JSON schemas after confirming that the current early implementations are aligned with them.

That said, JSON schemas alone are not enough since most operations have their own predefined socket types and ids. Suggestions on the machine-readable node spec representations are welcome.

[javagl]

It could be something simple as a starter....

output-2025-01-21.json

One could/should probably group this based on the Section structure, maybe even include the headers like ===== Cross Product or so. One could also think about parsing and understanding the types...

[javagl]

What is the timeline for this and its review (beyond "as quickly as possible")? I have some smaller "pending" review comments, but am not sure when I find the time to continue. Is the preferred way "one complete review", or "comments as soon as they come up"?

[lexaknyazev]

@javagl Feel free to post comments as soon as they come up.

[javagl]

OK, some minor comments inlined (I hope that I can continue here soon)

[javagl]

It's not "hard to predict", but rather "not always possible to predict" (I.e. semidecidable)

[javagl]

These nodes all have the name math/transform. This will cause trouble and ambiguities. They should be renamed to math/transform2, 3, 4, aligned with the naming of math/combineX.

The fact that the nodes and their types cannot be "templated" (despite what is suggested in math/transpose...), and the fact that nodes like math/sub exist once for int and once for "all float-based types" are a much broader issues with the type system, beyond the scope of a review comment)

[lexaknyazev]

This will cause trouble and ambiguities.

Please elaborate, i.e., propose an example graph that would be ambiguous but still conforming to the spec.

[javagl]

There currently does not seem to be a reasonable way to create a graph to begin with (except for a text editor)

The ambiguity mainly refers to the definitions and implementations of the nodes, and the authoring process, and maybe the validation. When there is a math/transform node, then it is not clear which type of node this is.

While the validation section contains a statement

(6.7, (7))
If the set of input value sockets defined by the declaration referenced by the declaration property does not match the set of input value sockets defined by the values property with regard to socket ids and types, reject the graph.

the declaration does not carry sufficient type information. Using math/transform with a float2 and a matrix3x3 is not valid, but it is not made explicit.

I know, this is one of the "back to the drawing board" comments, and it's unlikely to be addressed. So feel free to mark it as "resolved". (I'm looking forward to see how the authoring/tooling support will handle all this, eventually).