[Release v0.56 blocking] Investigate Resnet50 regressions in both TG + TGG model perf pipelines

[tt-rkim]

We were good until...

Up till this (LKG): https://github.com/tenstorrent/tt-metal/actions/runs/13456349605

the resnet model perf test was stable.

Occasionally the test_perf version with no trace or cqs would fail, but it was non-deterministic. Example: https://github.com/tenstorrent/tt-metal/actions/runs/13447053336/job/37574706436#step:10:113

But then... (2 weeks ago - afternoon of March 5)

Starting around the night of February 21st, we saw the perf trace 2cqs version hanging:

https://github.com/tenstorrent/tt-metal/actions/runs/13467326473/job/37635776939#step:10:119

This seems deterministic.

In fact, here are some runs which show the same deterministic hang:

• https://github.com/tenstorrent/tt-metal/actions/runs/13681043030/job/38254982193 at ~11am EST March 5: db0198e

The commit range is:

• d9263f2
• 5c8cbd2

But even more! ... (Afternoon of March 5 - now)

• Multiple tests started hanging around here: This pipeline ran at 8:30pm EST on March 5 https://github.com/tenstorrent/tt-metal/actions/runs/13688063291/job/38276072603#step:10:122 on a UMD bump commit from 4:30pm EST on March 5: https://github.com/tenstorrent/tt-metal/actions/runs/13688063291/job/38276072603#step:10:122
• Pipeline doesn't even start running: https://github.com/tenstorrent/tt-metal/actions/runs/13698185715/job/38305556323 around 8:30am on March 6

At 12:15pm, March 6, I did a reboot of the modules on the perf TGG machine via the Galaxy Management UI. Gonna see if that helps stamp out any additional issues so we can continue looking at the previous issue.

Doing another run here: https://github.com/tenstorrent/tt-metal/actions/runs/13704106394

cc: @tenstorrent/metalium-developers-infra

[mywoodstock]

OK, confirmed that the run started hanging starting with this commit: 1eef336
The commit before this (a7fffd259566503e5de2fdbaa335dc4c5ed524ce) is good.
cc: @yugaoTT

[yugaoTT]

@mywoodstock do we have a equivalent resnet50 on N150 CI? If that one doesn't hang, I don't see why TG hangs, since it's just a change within matmul op?

[mywoodstock]

@yugaoTT Yes, we have other tests with the same version of Resnet50, only this specific case of trace + 2cq on TG and TGG hang deterministically. All others are good.
I tried the current main with your commit reverted, and that does not hang.
So definitely something in the commit is causing the hang -- Not sure what can cause this though.