Code

added qlora + fsdp#14

Code

Merged

drisspg merged 19 commits intodrisspg:maindrisspg/transformer_nuggets:mainfrom weifengpy:fsdpweifengpy/transformer_nuggets:fsdpCopy head branch name to clipboardFeb 1, 2024

+271-14

Commits

Commits on Jan 31, 2024

• avoid import dynamic linear

  Show description for 74dc122
  weifengpy

  committed
  74dc122

  Copy full SHA for 74dc122

• fix fp8 import

  Show description for d5022ba
  weifengpy

  committed
  d5022ba

  Copy full SHA for d5022ba

• added qlora finetuning on fast-gpt model

  Show description for 9b2dca2
  weifengpy

  committed
  9b2dca2

  Copy full SHA for 9b2dca2

• add enable_qlora in train.py

  Show description for d113f06
  weifengpy

  committed
  d113f06

  Copy full SHA for d113f06

• add qlora to README

  Show description for 4347e48
  weifengpy

  committed
  4347e48

  Copy full SHA for 4347e48

• reorder QLoraMLP.w123 and use Hyperparameters

  Show description for 93acee0
  weifengpy

  committed
  93acee0

  Copy full SHA for 93acee0

• use separate finetune.py script from train.py

  Show description for db918e6
  weifengpy

  committed
  db918e6

  Copy full SHA for db918e6

• use qlora prefix in log files

  Show description for bb3b9ff
  weifengpy

  committed
  bb3b9ff

  Copy full SHA for bb3b9ff

• remove qlora import from train.py

  Show description for d65c3a0
  weifengpy

  committed
  d65c3a0

  Copy full SHA for d65c3a0

• fix linter

  Show description for 81b420d
  weifengpy

  committed
  81b420d

  Copy full SHA for 81b420d

• added FSDP + qlora

  Show description for a6093b8
  weifengpy

  committed
  a6093b8

  Copy full SHA for a6093b8

• remove duplicate code with torchao benchmark

  Show description for 5ab3a76
  weifengpy

  committed
  5ab3a76

  Copy full SHA for 5ab3a76

• remove duplicate code with torchao benchmark

  Show description for ad73696
  weifengpy

  committed
  ad73696

  Copy full SHA for ad73696

• reuse hyper param and train config

  Show description for 99ce030
  weifengpy

  committed
  99ce030

  Copy full SHA for 99ce030

Commits on Feb 1, 2024

• mark copied code

  Show description for 9fd3ac5
  weifengpy

  committed
  9fd3ac5

  Copy full SHA for 9fd3ac5

• Merge pull request #1 from weifengpy/fsdp_weif

  Show description for b99d3b1
  weifengpy

  authored
  b99d3b1

  Copy full SHA for b99d3b1

• Merge branch 'main' into fsdp

  weifengpy

  authored
  da88596

  Copy full SHA for da88596

• fix linter and log every 10 iter

  Show description for 6ae68cd
  weifengpy

  committed
  6ae68cd

  Copy full SHA for 6ae68cd

• use logging.info instead of print

  Show description for 7a0114d
  weifengpy

  committed
  7a0114d

  Copy full SHA for 7a0114d