I wrote a Tree-sitter grammar for Djot

[treeman]

Things happened, and I ended up creating a Tree-sitter grammar for Djot: https://github.com/treeman/tree-sitter-djot

It should be feature complete with Djot as described by the syntax description, but I'd appreciate any bug reports and feedback. This is the first Tree-sitter grammar I've created, so I'm sure I'm missing some tricks.

I also wrote a large-ish blog post on how to make a Tree-sitter grammar for a subset of Djot: https://www.jonashietala.se/blog/2024/03/19/lets_create_a_tree-sitter_grammar/

[NoahTheDuke]

This is cool as hell!

[rtbs-dev]

Ooh! Would you be willing to PR for the helix editor? They include a lot of TS grammar's by default, adding this one would be awesome! 🙏

[NoahTheDuke]

I don't use and don't know anything about the Helix editor so I'm the wrong person to ask, but adding this to nvim-treesitter has been pretty easy so I expect adding it to Helix would be equally easy.

[treeman]

I also know nothing about the Helix editor, but looking at their repo it indeed doeesn't seem difficult. A merge request needs to:

1. Add it to languages.toml: https://github.com/helix-editor/helix/blob/master/languages.toml
2. Insert the queries here: https://github.com/helix-editor/helix/tree/master/runtime/queries

Basically the same as for Neovim.

It's great to be enthusiastic, but the query files are still a bit immature, so I don't think we need to rush too much.

Still, if anyone wants to do it then feel free. I opened an issue for it on the grammar's repo to track it: treeman/tree-sitter-djot#10

[bdarcus]

I see the nvim-tree-sitter folks merged it. Looks good on quick testing!

[xxxserxxx]

I also know nothing about the Helix editor, but looking at their repo it indeed doeesn't seem difficult.

This doesn't answer the PR question, but I did write a blog entry about adding a tree sitter to Helix as a plain user. Things have moved a bit since, making it easier, and especially your tree sitter went swimmingly. In particular, helix now works in the user directory and all of the faffing about with permissions isn't necessary.

It's really only a few lines in the user's languages.toml and couple of commands on the command line -- assuming you have Rust & git and all that already installed:

1. Add this to your ~/.config/helix/languages.toml file (create it if you don't already have one):

   [[language]]
   name="djot"
   file-types=["dj"]
   auto-format=true
   scope="text.dj"
   roots=[]
   
   [[grammar]]
   name="djot"
   source={git="https://github.com/treeman/tree-sitter-djot", rev="0e9a836ec47612ade15645fb1680adb549894a6c"}
   

2. Run helix -g fetch
3. Run helix -g build
4. Symlink the grammars directory s.t. Helix can find them: ln -s ~/.config/helix/runtime/grammars/sources/djot/queries ~/.config/helix/runtime/queries/djot
5. Run helix --health languages and you should see djot; :

   $ hx --health languages | rg '^(Language|djot)'
   Language     LSP          DAP          Formatter    Highlight    Textobject   Indent
   djot         None         None         None         ✓            ✓            ✓
   

I'm sure there's a way to define the source without hard-coding a commit revision, but you can't leave the rev out and I haven't spent any time trying to figure it out. Anyway, it worked beautifully for me.

[jgm]

Amazing! I'm starting an Emacs major mode that uses this.

[zetashift]

Hi, thank you for Djot! I was wondering if you ever ended up making the djot-ts-mode?

[jgm]

I have a work in progress. It's fairly basic.
https://gist.github.com/jgm/17e17d6730efc253078b23e12aefd3df

[vassudanagunta]

I'm curious how it performs against the manually coded implementations. Have you done any comparisons?

Also, does it make sense to either add your Tree-sitter grammar, or C and WASM parsers generated from it, to the above list?

[treeman]

I haven't done any performance comparisons against other implementations. I've only noted that it's fast enough for decently large files for my use-case.

Maybe it would make sense to add it to the implementations? But we should keep in mind that this is a Tree-sitter grammar, and should not be relied on to for instance generate html.

[vassudanagunta]

You mean to say it should only be used to implement syntax highlighting or to support syntax-aware editors?

If so, is it because renderers should be built against djot's standard AST, not the syntax tree produced by this Tree-sitter grammar? Does djot even have a standard AST, across the Lua, Javascript, Rust, Go, etc implementations?

Or is it because Tree-sitter parsers produce concrete syntax trees, not abstract syntax trees, more suitable for syntax highlighting and editors, not so suitable for compilers, interpreters or renderers?

[treeman]

Uh, I realize I'm out of my depth here and I should've said that I've heard that dedicated parsers are better suited, but I'm no expert on the trade-offs here. hides

Does djot even have a standard AST, across the Lua, Javascript, Rust, Go, etc implementations?

I don't think so no, at least with jotdown you don't have an AST, only a stream of iterators to work with.

[vassudanagunta]

Fair enough. I'm out of my depth too, wrt to Tree-sitter, hence the questions. I'm working on a metalanguage called Plain Text Style Sheets which focuses on human oriented languages like Markdown. Parsers based on it would output a standard abstract grammar tree or semantic tree (I'm not sure what the right terms are).