smart punctuation as a filter, and filter stdlib

[wti]

(I love the possibilities for djot. I've built many tools based on markdown using pandoc. I appreciate the design goals and the reliability of the implementation, and I'm grateful for all you and those you inspire have done.)

I'm confused though by making smart punctuation required. The overarching principle of markup should be that it doesn't change content. I don't want magically my headings capitalized, or periods after my list items, or my straight quotes made curly, or my dots made into ellipses, even if those are all good things. I want markup to specify style and structure - only.

It's partly as a user, but also as a developer using markdown to drive both content- and data-driven applications. I have to be able to validate that what went in is what came out. I would have to replicate any content-changing logic, and the smarter it is (e.g., about curly quotes), the more logic I have to duplicate. (Being forced from ascii into unicode for data-driven apps is also lamentable.)

Instead, smart punctuation would make a great standard filter that users could opt-in to, if needed in their context.

More generally, many design pressures on djot could be relieved through a process for standardizing filters such that users can rely on them with minimal declaration or configuration. Design proponents can be asked to work up proposals as filters for community experiment. Depending on the feedback, filters would then be available third-party, pulled into a djot stdlib, or (rarely) pulled into djot itself. The stdlib filters could offer support for other filters (e.g., with attribute processing) or offer high-traffic changes to style/structure, generated content sections, or content changes like smart punctuation. (Sorry to elaborate -- I'm sure you've considered all this.)

[jgm]

One reason smart punctuation is handled in the main parser is that it's actually very tricky to match up quotes properly. Converting --- to em-dash or ... to ellipses could easily be done in a filter, but quotes are harder.

But I do see that this is something one might want to be able to turn off.

A filter that goes in the reverse direction -- "de-smartifying" -- is already possible; that's a very straightforward transformation. Just convert

double_quoted
  CONTENT

to

span
  str "\""
  CONTENT
  str "\""

and similarly for ellipses and dashes.

[wti]

Ah, of course - great. I can dumb content back down today with filters.

I see the value of quote-matching and that it would be hard in filters, driving it into the parser.