cabal-docspec: Fails to detect examples with non-standard unicode characters present

[amesgen]

E.g. this example is not detected:

-- >>> length "猫"
-- 1

This is due to haskell-lexer not supporting this:

 Λ L.lexerPass0 "-- >>> length \"猫\"\n"
[(Commentstart,(Pos {char = 0, line = 1, column = 1},"--")),(ErrorToken,(Pos {char = 2, line = 1, column = 3}," >>> length \"")),(TheRest,(Pos {char = 15, line = 1, column = 16},"\29483\"\n"))]

This could be fixed in haskell-lexer (but maybe this actually is accurate behavior for Haskell 98?), but as stubbornPass0 already works around another haskell-lexer deficiency, it might be acceptable to do so again in this case? See #132 for that.

Minimal reproducible example

https://github.com/amesgen/stuff/tree/cabal-docspec-unicode

 $ git clone https://github.com/amesgen/stuff --single-branch --branch cabal-docspec-unicode
 $ cd stuf && cabal build
 $ cabal-docspec
Total:         0; Tried:    0; Skipped:    0; Success:    0; Errors:    0; Failures    0
Examples:      0; Tried:    0; Skipped:    0; Success:    0; Errors:    0; Failures    0

[LeventErkok]

@amesgen Just ran into this very same issue.

Are there any workarounds, or fixes planned?

[amesgen]

The workaround in #132 (just rebased) still works fine for me, but it is rather ad-hoc.

[LeventErkok]

Thanks @amesgen

This doesn't work for me, unfortunately. My code looks like:

data SStroke

-- | The sheffer stroke operator
(︱) :: SStroke -> SStroke -> SStroke
(︱) = uninterpret "︱"
infixl 7 ︱

-- | Negation in terms of ǀ
ﬧ :: SStroke -> SStroke
ﬧ x = x ︱x

It appears the definition for ﬧ is throwing things off.

Do you think this is a separate issue?

[amesgen]

Yeah, that ﬧ being in the code (and not just in a comment), so the workaround in #132 indeed doesn't apply. It's still the same general issue with haskell-lexer though, and maybe it can be worked around in a similar way as #132, but it is probably more appealing to properly fix haskell-lexer at this point.

[LeventErkok]

@amesgen I filed an issue against haskell-lexer; looks like Iavor is maintaining it and hopefully he can get a fix put in.

Do you have a patch you can submit against it? Or commenting on the ticket regarding the details might help Iavor get to the issue quicker. Thanks!

[LeventErkok]

@amesgen

Looks like Iavor is making progress. But there's another issue: Shouldn't doctest complain when lexing fails? Right now it appears it simply ignores those files (or portions of it?) that it cannot lex.

Should we be filing another issue where these failures cause erroring out? Otherwise, a user might think things are getting tested, which are just not run because of some lexing/parsing issue kicks in.