nemo-automodel: fsdp2 support for peft #12008
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BoxiangW
reviewed
Jan 31, 2025
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
from
5948ab7 to
8721756
Compare
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
from
b21a5ff to
de34b2f
Compare
akoumpa
requested review from
pablo-garay,
ko3n1g,
thomasdhc and
chtruong814
as code owners
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
4 times, most recently
from
70865f5 to
3665444
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
3 times, most recently
from
8d222a4 to
a81e480
Compare
akoumpa
changed the title
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
from
d337aa8 to
e66d469
Compare
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
from
04eaaf0 to
ba03e8e
Compare
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
from
da1bd02 to
c4bc136
Compare
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
from
87a97c7 to
0731898
Compare
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
from
1cbd6f6 to
1c088eb
Compare
akoumpa
force-pushed
the
akoumparouli/automodel_fsdp2_peft_fix
branch
from
feb6136 to
65c7b20
Compare
Signed-off-by: akoumpa <akoumpa@users.noreply.github.com>
ko3n1g
approved these changes
Feb 4, 2025
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #12008 +/- ##
==========================================
- Coverage 30.30% 30.30% -0.01%
==========================================
Files 1387 1387
Lines 176283 176293 +10
Branches 27091 27096 +5
==========================================
- Hits 53422 53420 -2
- Misses 118775 118788 +13
+ Partials 4086 4085 -1 ☔ View full report in Codecov by Sentry. |
|
[🤖]: Hi @akoumpa 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully So it might be time to merge this PR or get some approvals I'm just a bot so I'll leave it you what to do next. //cc @pablo-garay @ko3n1g |
BoxiangW
added a commit
that referenced
this pull request
Feb 7, 2025
* Use patch_linear_module for FSDP2 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Use patch_linear_module for FSDP2 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add fsdp2 strategy to test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add --num-nodes option Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add --num-nodes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * minor fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add to_cpu in utils Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use to_cpu Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * shard adapter weights Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use to_cpu from utils Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use get_automodel_from_trainer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove some mcore logic from save_checkpoint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * call to_cpu in strategy's save_checkpoint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix ckpt saving Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove unused import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add missing import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * simplify Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * minor fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix :/ Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix :/ Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix :/ Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix :/ Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * # noqa: F821 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * noqa Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com>
youngeunkwon0405
pushed a commit
to youngeunkwon0405/NeMo
that referenced
this pull request
Feb 10, 2025
* Use patch_linear_module for FSDP2 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Use patch_linear_module for FSDP2 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add fsdp2 strategy to test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add --num-nodes option Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add --num-nodes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * minor fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add to_cpu in utils Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use to_cpu Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * shard adapter weights Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use to_cpu from utils Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use get_automodel_from_trainer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove some mcore logic from save_checkpoint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * call to_cpu in strategy's save_checkpoint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix ckpt saving Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove unused import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add missing import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * simplify Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * minor fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix :/ Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix :/ Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix :/ Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pylint fix :/ Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * # noqa: F821 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * noqa Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Adds support for fsdp2 + peft.
Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information