Add check endpoint which can be used with nginx' auth_request function

[SuperSandro2000]

This allows to use annubis via nginx' auth_request function.

This is a simplified example extracted from my config:

location /.within.website/ {
        proxy_pass http://127.0.0.1:8923;
        auth_request off;
}
location @redirectToAuth2ProxyLogin {
        return 307 /.within.website/?redir=$scheme://$host$request_uri;
        auth_request off;
}
auth_request /.within.website/x/cmd/anubis/api/check;
error_page 401 = @redirectToAuth2ProxyLogin;

[baconwaifu]

If this had a way to return both an unauthorized status and the response body (I see no reason why the original path has to return a 200; cloudflare returns 403 when it intercepts with the challenge) then this would work rather seamlessly with traefik's ForwardAuth middleware as well.

[Xe]

(I see no reason why the original path has to return a 200; cloudflare returns 403 when it intercepts with the challenge)

I hate to inform you that there is a reason to it, and the rationale kinda incredibly sucks. If AI scrapers see 4xx statuses, they just re-queue the page for trying again later. The only way to get them to actually go away and stay away is to lie and return HTTP 200 when the status is not semantically 200. This is surely a violation of some RFC somewhere, but sadly this is what works.

[baconwaifu]

I hate to inform you that there is a reason to it, and the rationale kinda incredibly sucks. If AI scrapers see 4xx statuses, they just re-queue the page for trying again later. The only way to get them to actually go away and stay away is to lie and return HTTP 200 when the status is not semantically 200. This is surely a violation of some RFC somewhere, but sadly this is what works.

That is utterly horrifying. And also just plain stupid on the part of the bots, considering that most non-ratelimit responses probably aren't going to change from time alone...

So that rules out changing the default proxy-mode response code. The other alternative is to have the webhook mode configurable to also return the challenge body, since Traefik expects the webhook to return the error page directly, rather than having an easy way to do an error_page like nginx can (or it can supply a redirect to the magic prefix, as 3XX also works to hijack the response)

[Xe]

That is utterly horrifying. And also just plain stupid on the part of the bots, considering that most non-ratelimit responses probably aren't going to change from time alone.

Yep! This is the reality we got. The really annoying part is that this makes the user experience super bad, but this was not a decision I made out of choice. It is a decision made out of desperation. Amazonbot in particular speeds up its requests if you return a 5xx response.

[Xe]

@SuperSandro2000 I'm going to manually recreate this PR in TecharoHQ/anubis as I am moving the project there (see #698 and TecharoHQ/anubis#1).

Seriously though, thank you for the contribution! It's what actually pushed me over the line to want to move this to a dedicated org/repo.

[SuperSandro2000]

😅 if that was all it took, but seriously back: I debugged this until 5 am or so because matching nginx absolute and relative paths with auth_request and adding the right return codes to anubis for this was also not as easy as I would have liked it.

My time it currently a bit spotty but feel free to refactor/clean up/whatever this. I try to keep an eye on it but no promisses.