Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Release v0.56 blocking] Investigate possible TG Falcon7b model regression #18717

Closed
tt-rkim opened this issue Mar 6, 2025 · 2 comments
Closed
Assignees
Labels
ci-bug bugs found in CI LLM_bug P0

Comments

@tt-rkim
Copy link
Collaborator

tt-rkim commented Mar 6, 2025

Setting P0 because this is blocking v0.56.

Falcon7b demo on TG was last good here:

https://github.com/tenstorrent/tt-metal/actions/runs/13681043030/job/38254977872
on
db0198e

but we now have what looks like a hang:

https://github.com/tenstorrent/tt-metal/actions/runs/13689206984/job/38279608916
on
2bbd8ca

However, there were a bunch of TG hangs, so this may have solved it: #18702

Running now: https://github.com/tenstorrent/tt-metal/actions/runs/13703860358

cc: @tenstorrent/metalium-developers-infra

@tt-rkim tt-rkim added ci-bug bugs found in CI LLM_bug P0 labels Mar 6, 2025
@tt-rkim tt-rkim changed the title [Release v0.56 blocking] Investigate TG Falcon7b model regression [Release v0.56 blocking] Investigate possible TG Falcon7b model regression Mar 6, 2025
@tt-rkim
Copy link
Collaborator Author

tt-rkim commented Mar 6, 2025

There is another regression on top of the one above, but it's a non-deterministic slice error described here: #15059

This shows ND: https://github.com/tenstorrent/tt-metal/actions/runs/13634038681

@skhorasganiTT
Copy link
Contributor

The job you linked passed @tt-rkim, can this be closed?

@tt-rkim tt-rkim closed this as completed Mar 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-bug bugs found in CI LLM_bug P0
Projects
None yet
Development

No branches or pull requests

2 participants