Kernel profiler based NoC event tracing

[bgrady-tt]

Problem Description

This PR adds support for recording detailed traces of all NoC activity initiated by Tensix worker cores. This data is then written to a JSON format file. These JSON traces can be either analyzed directly (as a sort of log file) or consumed by either flows (e.g. used to verify software noc performance estimator).

What's Changed

NoC tracing builds on top of the timestamped data packets (PacketType::TS_DATA) feature in the kernel profiler.

NoC tracing can be enabled on top of normal kernel profiling by setting the ENV variable TT_METAL_DEVICE_PROFILER_NOC_EVENTS=1.

A timestamped packet is recorded on each worker core for each call to dataflow_api.h:noc_async_* functions. All arguments to the dataflow_api.h function are bit-packed into the TS_DATA payload. This is done by adding new stripped-by-default macros to each noc_async* call of interest in dataflow_api.h.

  // example of RECORD macro within dataflow_api.h
  template <bool enable_noc_tracing = true>
  inline void noc_async_read(...) {
      // if TT_METAL_DEVICE_PROFILER_NOC_EVENTS != 1, the following macro is stripped out
      RECORD_NOC_EVENT_WITH_ADDR(...);
      // noc_async_read continues unmodified here ... 
  }

For situations where it is impossible/undesirable for noc_async_* calls to be profiled, template arguments have been added to all effected sites so that noc tracing can be disabled at the call site. This is done within the kernel profiler itself to avoid a circular dependency.

  noc_async_write<false>(...); // noc tracing is stripped out at compile time at this call site

Concerns

• kernel_profiler::quick_push() and supporting code has been modified to compile for BRISC and NCRISC, even if dispatch is false.
• kernel_profiler::timestampedData() has been modified to check if bufferHasRoom; if there is no room, quick_push() is automatically called to flush the buffer to make space.
• Considerable overlap with existing features like Watcher Waypoints and NoC address sanitizer.

As far as I can tell (informal testing), regular kernel profiling still works as expected (with or without noc event profiling being enabled).

Despite the overlap with existing profiler-based features, there is meaningful differentiation here. Capturing all of the details of each NoC transaction (packet size, virtual channel, timestamp) is useful (probably to many different teams) and currently not accomplished by other methods.

Checklist

• Post commit CI passes
  • Post commit CI passes before rebase to latest origin/main

[mo-tenstorrent]

Thanks for the clean up on logging functions.

Could you please make sure the following are green:

Device Perf: https://github.com/tenstorrent/tt-metal/actions/workflows/perf-device-models.yaml

uBenchmark:https://github.com/tenstorrent/tt-metal/actions/workflows/metal-run-microbenchmarks.yaml

multi device (T3K) profiling: https://github.com/tenstorrent/tt-metal/actions/workflows/t3000-profiler-tests.yaml

[mo-tenstorrent]

Nice !! Thanks.

[mo-tenstorrent]

Depending on how get_type_name is implemented for the op, we might acquire considerable perf hit on cached runs.

Can we push the emplace to right before the return and lookup opName from cached_ops. It will be filled by the end of this function. We do need a local return variable this way.

[mo-tenstorrent]

I think if getPhysicalAddressFromVirtual takes device_id instead of IDevice*. We don't need to pass IDevice down from readRiscProfilerResults