Torchlight syntax highlighting

[jorqensen]

I suggest making Torchlight an optional choice for code block syntax highlighting.

Right now, there is currently one theme of choice when using Retype. Granted this is not "bad" I think with the planned changes for variables etc. we see a future of more customization to Retype, so this would be a nice addition.

Furthermore, Torchlight introduces proper syntax highlighting for all languages.

Torchlight has a pricing for commercial use. However, if it's up to the user of Retype to provide their own API key and having Torchlight being an optional choice, I assume this could be prevented?

Configuration could be something along the lines of this in retype.yml

snippets:
  torchlight: true
  torchlightKey: some-api-key-here

(Following the current project configuration it would either make sense to throw it under snippets or integrations).

This however will expose the API key since retype.yml is comitted to a git repo usually. Would it perhaps be neccesary to implement the ability to load .env vars into the retype.yml file at this point? Or would this very quickly add a lot of complexity for something that should be simple?

[geoffreymcgill]

Torchlight would be a challenging integration for us to implement.

As Torchlight is an external service that would need to be integrated in the build process, the Retype App would need to make external calls to the Torchlight service in order to fetch each formatted code block. That would significantly slow down the build process. That process would also break a fundamental goal of Retype that the app does not (cannot) make external calls. Everything needs to be self contained within the Retype App.

There are options, but it would take a good bit of work to implement any of these:

1. Build a Retype component parsing API with event hooks

A component beforewrite event handler that could be built to pass the parsed Markdown to an external service instead of Retype converting the Markdown. Essentially custom Markdown Converters.

For instance, while parsing, if Retype found a code block component, the code within that code block would be sent to Torchlight and Torchlight would return the final HTML that Retype would then inject into the finished document, instead of Retype generating the Prism syntax highlighter requirements.

This is possible and the most flexible, but it would be a major project and take time to implement. Time and human resources that we do not have right now.

Hitting the max 5000 requests per month limit of Torchlight seems like it would be very easy to do with that kind of Retype integration. Of course we could try building in some kind of caching mechanism to avoid sending code blocks that had already been previously compiled and had not changed, but that brings even more complexity.

Good:

• Flexibility. Not only Torchlight could be supported. Generic Converters and event handlers would allow unlimited flexibility and customization of the final generated HTML for any Markdown component.

Bad:

• Lots of complex parts would need to be built into Retype.
• For Torchlight, it would mean an external service calls in order to just build your project.
• Build performance would be slow because of external calls.
• Likely would require Torchlight commercial license.

2. Retype integrates the same VS Code code parsing component as Torchlight

Basically this means building our own Torchlight like service, but integrated directly into Retype. Use the same tech as Torchlight but not calling an external service.

Good:

• Performance would be fantastic.
• The component parsing event lifecycle as proposed above would not have to be built, so saving time and complexity.
• No Torchlight subscription would be required.
• Functionality would be completely built into Retype and replace the Prism syntax highlighting.

Bad:

• Would just take time to build.
• Prism JS syntax highlighting is doing a good job. Obviously could be better, but it's not that bad right at the moment.

I'm sure that are more scenarios and not just these two ideas. I'll keep this thread updated with other options as they come to me.

Those are just a few of my thoughts on Torchlight integration. Certainly not saying no, but just doesn't seem feasible or necessary at the moment.

Thoughts?

[DannyBen]

Thoughts?

I am mostly concerned about retype becoming a bloatware sooner than expected...
Big, complex moving part are a baggage you carry with you everywhere.

On the other hand - in my view, Retype is doing one thing, and it should aim to do it the best it can. As a documentation service, it is a good objective to strive to want to show the best syntax highlighting possible. If this can be achieved in a local fashion (even if it might be slightly less sophisticated), it could be enough, if not, then perhaps something like torchlight should be considered.

For the sake of exploration, let's assume you do decide you want torchlight, how about this approach:

1. You make it a design guideline, to keep the base retype package, clean, lean and mean. No niche functionality in it, only mainstream core package.
2. You implement multiple hooks, similar to git hooks, or hooks in a testing framework: before_all, before_each, after_render etc.
3. This hooks mechanism is built in a generic way: It calls an executable with parameters, or passes parameters in environment variables - as long as the interface is clear, simple, and stable.
4. This - in effect - makes retype pluggable. Now, anyone - not just the retype team - can develop plugins, which are executable (in any language) that adhere to the plugin guidelines.
5. If and when the retype team has time to develop "official" plugins, they are deployed as a separate package.

This will allow anyone (Retype team or not) to develop a Torchlight plugin:

# retype.yml
hooks:
  before_each: retype-torchlight.rb

# retype-torchlight.rb - pseudocode
file = ENV['RETYPE_FILE']
content = File.read file
if file.contains_code?
  content = torchlight_convert content
end

# send output back to STDOUT, as part of the retype hooks contract (as an example)
puts content