Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create logging proxy #653

Merged
merged 13 commits into from
Dec 19, 2023
Merged

feat: create logging proxy #653

merged 13 commits into from
Dec 19, 2023

Conversation

emizzle
Copy link
Contributor

@emizzle emizzle commented Dec 14, 2023

Add a logging proxy to achieve several goals:

  1. Fix json logging output (run with --log-format=json) which was effectively broken for many types.
  2. Ability to specify log output for textlines and json sinks together or separately
    • This is useful if consuming json in some kind of log parser and need valid json with real values
    • eg a shortened Cid is nice to see in a text log in stdout, but won't provide a real Cid when parsed in json
  3. Remove usages of nim-json-serialization from the codebase
  4. Remove need to declare writeValue for new types
  5. Remove need to avoid importing or exporting toJson, %, %* to prevent conflicts

When declaring a new type, one should consider importing the codex/logutils module, and specifying formatIt. If textlines log output and json log output need to be different, overload formatIt appropriately. If a json serialization is needed, it can be declared with a % proc. Only codex/logutils needs to be imported.

With this PR in:

  • Instead of importing pkg/chronicles, import pkg/codex/logutils
    • most of chronicles is exported by logutils
  • Instead of importing std/json, import pkg/codex/utils/json
    • std/json is exported by utils/json which is exported by logutils
  • Instead of importing pkg/nim-json-serialization, import pkg/codex/utils/json
    • one of the goals is to remove the use of nim-json-serialization
import pkg/codex/logutils

type
  BlockAddress* = object
    case leaf*: bool
    of true:
      treeCid* {.serialize.}: Cid
      index* {.serialize.}: Natural
    else:
      cid* {.serialize.}: Cid

logutils.formatIt(LogFormat.textLines, BlockAddress):
  if it.leaf:
    "treeCid: " & shortLog($it.treeCid) & ", index: " & $it.index
  else:
    "cid: " & shortLog($it.cid)

logutils.formatIt(LogFormat.json, BlockAddress): %it

# chronicles textlines output
TRC test                                       tid=14397405 ba="treeCid: zb2*fndjU1, index: 0"
# chronicles json output
{"lvl":"TRC","msg":"test","tid":14397405,"ba":{"treeCid":"zb2rhgsDE16rLtbwTFeNKbdSobtKiWdjJPvKEuPgrQAfndjU1","index":0}}

In this case, BlockAddress is just an object, so utils/json can handle serializing it without issue (only fields annotated with {.serialize.} will serialize (opt-in).

If one so wished, another option for the textlines log output, would be to simply toString the serialised json:

logutils.formatIt(LogFormat.textLines, BlockAddress): $ %it
# or, more succinctly:
logutils.formatIt(LogFormat.textLines, BlockAddress): it.toJson

In that case, both the textlines and json sinks would have the same output, so we could reduce this even further by not specifying a LogFormat:

type
  BlockAddress* = object
    case leaf*: bool
    of true:
      treeCid* {.serialize.}: Cid
      index* {.serialize.}: Natural
    else:
      cid* {.serialize.}: Cid

logutils.formatIt(BlockAddress): %it

# chronicles textlines output
TRC test                                       tid=14400673 ba="{\"treeCid\":\"zb2rhgsDE16rLtbwTFeNKbdSobtKiWdjJPvKEuPgrQAfndjU1\",\"index\":0}"
# chronicles json output
{"lvl":"TRC","msg":"test","tid":14400673,"ba":{"treeCid":"zb2rhgsDE16rLtbwTFeNKbdSobtKiWdjJPvKEuPgrQAfndjU1","index":0}}

Note

Special thanks to @elcritch for the help on this one!

@dryajov
Copy link
Contributor

dryajov commented Dec 14, 2023

Very nice! Thanks, I'll take a closer look, but looks like a great refactor!

@elcritch
Copy link
Contributor

Wow nice you got it up! I'll review after I eat lunch. :)

@emizzle emizzle force-pushed the feat/logging-proxy branch 2 times, most recently from 808099f to ae9b0cb Compare December 19, 2023 01:30
@emizzle
Copy link
Contributor Author

emizzle commented Dec 19, 2023

@elcritch, thanks for the review! Apart from the question to @benbierens, all points have been addressed. Looking forward to your re-review 🙏

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.
We only need to import utils/json instead of std/json
Was causing unit tests to fail on Windows.
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
Both json and textlines formatIt were not needed, and could be combined into one formatIt
debug output and logformat of json for integration test logs
@emizzle emizzle enabled auto-merge (squash) December 19, 2023 21:42
@emizzle emizzle merged commit 27f585e into master Dec 19, 2023
8 checks passed
@emizzle emizzle deleted the feat/logging-proxy branch December 19, 2023 22:12
@emizzle emizzle restored the feat/logging-proxy branch December 20, 2023 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants