How to approach linking / scope with externally defined objects?

[BCarley]

Hello!

I am working through trying to redefine an existing "stateful rule" language with langium and would appreciate some advice on the best approach with respect to scopes.

Language Background

The aim of the language is to allow users to write rules that evaluate against passed in data. The user is able to store previous executions as variables to make the execution stateful. The passed data is defined using a json-schema document and so I aim to pass this to the language-server by a service.

Question

What is the best approach for scoping / validation? I would like to be able to use language server features for the user defined variables, scoped by rule and var but also suggest and infer types for data based on the json-schema service.

In the included grammar I have followed the QualifiedName approach given in the domain model. However, as data is not defined in the language itself, and so cannot be included in the ScopeComputation, (as far as I understand) I was not able to use linking.

There is an interesting example (hello-json by @goto40) that uses auto-completion and avoids linking, but I feel like this would be giving up all the built in linking functionality that makes langium so interesting.

I have avoided using keywords for scoping data as there might be some cases where the passed data causes a clash with some keyword. e.g. data.data > 2, but I would be interested to hear if this is the best approach.

Example Rule

// variable definition
var.var1: var.var1 + data.value * 2;

// rule referencing data directly
rule.rule1: data.value > 1;

// rule referencing a variable
rule.rule2: var.var1 > 2;

Grammar

grammar StatefulRules

entry Model:
    (declarations+=Declaration)*;

// <type>.<name>: <expression>;
Declaration:
    type=ID '.' name=ID ':' statement=Stmt ";";

Stmt:
    Expression (EOL)?;

Expression:
    Addition;

Addition infers Expression:
    Multiplication ({infer BinaryExpression.left=current} operator=('+' | '-') right=Multiplication)*;

Multiplication infers Expression:
    Logical ({infer BinaryExpression.left=current} operator=('*' | '/') right=Logical)*;

Logical infers Expression:
    Coalescing ({infer BinaryExpression.left=current} operator=('&&' | '||') right=Coalescing)*;

Coalescing infers Expression:
    Comparison ({infer BinaryExpression.left=current} operator='??' right=Comparison)*;

Comparison infers Expression:
    Primary ({infer BinaryExpression.left=current} operator=('<' | '<=' | '>' | '>=' | '==' | '!=') right=Primary)*;


Primary infers Expression:
    '(' Expression ')' |
    QualifiedName | 
    Literal;


Literal: StringLiteral | BooleanLiteral | PercentageLiteral | NumberLiteral;

PercentageLiteral:
    value=NUMBER "%";

NumberLiteral:
    value=NUMBER;

StringLiteral:
    value=STRING;

BooleanLiteral:
    value?='true' | 'false';

QualifiedName returns string:
    ID ("." ID)*;

// Ignore whitespace
hidden terminal WS: /\s+/;

// Identifier tokens,
terminal ID: /[a-zA-Z][a-zA-Z0-9_]*/;

// Used for end of statement
terminal EOL: /\\r?\\n/;

// literals
terminal NUMBER returns number: /\d+((\.\d+)?([eE][\-+]?\d+)?)?/;
terminal STRING returns string: /"(\\.|[^"\\])*"|'(\\.|[^'\\])*'/;

// Multi line and single comments
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;
hidden terminal ML_COMMENT: '/*' -> '*/';

[goto40]

I am looking forward to see some comments. I am also still thinking of how to combine different models (with different meta models). Even with the referenced hello-json example, you do not handle cases, when the json data is changed in the editor...

Langium is great an still evolving and I am looking forward to see what this very useful tool brings us in the future. Thank you to the langium team!

[BCarley]

@goto40 agreed, Langium seems really well designed and I feel like I'm only getting stuck because I'm trying to do something slightly unusual. I'll try and post any progress I make on a json-schema service too!

[msujew]

Hey @BCarley,

the general solution is usually to convert the externally defined objects into some AstNode instance. The best way would be to first define at least some interfaces (in your grammar) that cover whatever you want to do with those objects. The second step would either be to convert your objects into DSL code (and parse that into a document), or create a document from those objects directly. The latter one is less work but the former has the benefit of better editor integration. Then, just run those documents through the document builder life cycle and allow your actual DSL code to reference those elements via the normal cross reference resolution system.

That's the general, high level overview. That way, you stay inside of the existing scoping system, but allow to reference documents that are loaded somewhat dynamically from a different source. There's also this discussion that is somewhat related to this topic.

[goto40]

I just saw the updated docs describing multiple languages within one langium project (https://langium.org/guides/multiple-languages/). Good work!

I wonder if I could use your hints given above to "glue" two separate langium projects, allowing to reference elements in one dsl from the other... (Including live vscode updates while editing).

If you could give some concrete hints I could try something (maybe based on the persons or requirements example).

[msujew]

We often split a project into a user and definition language. Similar to how TypeScript has .ts and .d.ts files. One can only have definitions, while the other is actually used by the user to write their code. We use the same mechanism as outlined in the guide for this. Users can always reference elements of the definition language part, but the definition language itself is self-contained.

I cannot really share any work though, so the only hints I can give are more abstract. Once you have multiple languages in your project (the linked documentation should guide you through that), gluing two Langium projects together should be fairly trivial.

[BCarley]

Thank you for your feedback, this comment has really helped me understand a potential approach for the magicness of our language, without sacrficing the excellent built in lsp features.

I think I need 3 definitions:

• Rule language itself
• Plain JSON, which can be referenced by a ScopeProvider when calculating what fields are available under the data namespace, based on the JSON Schema document.
• Rule definitions, similar to the .d.ts files mentioned, to define the builtins for the language.

Thank you for updating the linked documentation, I will try and work through it.

[BCarley]

@msujew - is there a recommended approach for rewriting the scopes across multiple languages? I'm assuming this goes in the shared langium services, but I can't quite figure out how to get this to work / if this is a valid solution.

I am trying to include a scope service for the rule language that is able to call into the json language and grab a specific AST node. The other alternative I am looking at is trying to go through the services.shared.ServiceRegistry.

[msujew]

You probably make this more complicated than it needs to be. You can just create a single scope provider (that works for all languages) and bind it for each language module independently.

[BCarley]

For context, this is my first attempt at a .json parser, building on the example given in chevrotain.

// implement a simplistic JSON parser so we can provide AST nodes for the SchemaService
grammar JsonLangium

entry JsonRoot:
    value=JsonElement;

JsonObject:
  "{" entries+=JsonObjectEntry? ("," entries+=JsonObjectEntry)* "}";

JsonObjectEntry:
  key=JSON_STRING ":" value=JsonElement;

JsonArray:
  "[" items+=JsonElement? ("," items+=JsonElement)* "]";

JsonBooleanLiteral:
    value=("true" | "false");

JsonElement: JsonObject | JsonArray | JsonStringLiteral | JsonNumberLiteral | JsonNullLiteral | JsonBooleanLiteral;

JsonStringLiteral:
  value=JSON_NUMBER;

JsonNumberLiteral:
  value=JSON_NUMBER;

JsonNullLiteral:
    value="null";

hidden terminal JSON_WS: /\s+/;

terminal JSON_NUMBER returns number: /-?(0|[1-9]\d*)(\.\d+)?([eE][+-]?\d+)?/;

terminal JSON_STRING returns string: /"(?:[^\\"]|\\(?:[bfnrtv"\\/]|u[0-9a-fA-F]{4}))*"/;

I have prefixed everythnig with Json as I'm importing this into the main grammar as per the mutli-doc example.

[BCarley]

Question for Literal values i.e. string, number, boolean:

StringLiteral:
    value=STRING;

BooleanLiteral:
    value?='true' | 'false';

terminal STRING returns string: /"(\\.|[^"\\])*"|'(\\.|[^'\\])*'/;

My understanding is that we would want to link these to the builtin definitions via references so we can access the members for them, similar to python / javascript. Can this be done in the grammar itself or would we need to extend a service?

[msujew]

Can you give an example/a bit more context? Literals are supposed to be just that. I'm not sure how a boolean/number is supposed to be a reference.

[BCarley]

For example in python we can do:

"hello, world".split(",")

My, probably flawed, understanding is that we would need to be able make split available in the scope for the "hello, world" StringLiteral. I was planning on doing this by defining a String class in the definition language and linking this together.

[BCarley]

Apologies, I think I have answered my own question.

Copying the langium-lox example of the ScopeProvider (link here).

I could use an isStringType check, and then return a string definition from my definition document, using pretty much the same logic as langium-lox.

[msujew]

Ah, yes that's what you mean. Exactly, the type system would effectively return a special class type of your builtin library in case it encounters a string type/literal. Every property of that literal is then mapped to that class.

[ejfasting]

Hi @BCarley ,

Any chance you can share the code for implementation? :)
I'm working on something similiar, and it would be neat to see what you ended up with.