New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix normalize instability crash #139

Open

shahbuland wants to merge 1 commit into richzhang:master from shahbuland:patch-1

shahbuland commented Jan 30, 2025

Change eps to 1.0e-6 for bf16 stability and so sqrt is not given a negative number.

Before fix

Training runs randomly crash because radicand gets close to zero and is rounded to a small negative number
eps is rounded to 0 in mixed precision training runs
After fix
Denominator stays positive
Radicand stays non-negative


          Fix normalize instability crash

dbabf21

Change eps to 1.0e-6 for bf16 stability and so sqrt is not given a negative number.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet