Skip to content

Commit

Permalink
finally?
Browse files Browse the repository at this point in the history
  • Loading branch information
johnjaejunlee95 committed Feb 8, 2025
1 parent 139f3bc commit 2cbc546
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 27 deletions.
11 changes: 2 additions & 9 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,10 @@
source "https://rubygems.org"

# gemspec
gem "beautiful-jekyll-theme", "6.0.1"
gem "ffi", "= 1.16.3"
gem 'jemoji'
gem 'tzinfo'
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw]
gem 'wdm', '>= 0.1.0'
gem 'eventmachine', '1.2.7', git: 'https://github.com/eventmachine/eventmachine.git', tag: 'v1.2.7'
gem "jekyll-remote-theme"
gem 'rubyzip', '2.3.0'
gem 'sassc' #, '2.4.0'
gem "beautiful-jekyll-theme", "6.0.1"
gem 'html-pipeline'
gem 'selma', '~> 0.4.12'
gem 'zeitwerk', '~> 2.7', '>= 2.7.1'
gem "rouge"
gem 'eventmachine', '1.2.7', git: 'https://github.com/eventmachine/eventmachine.git', tag: 'v1.2.7'
29 changes: 11 additions & 18 deletions _posts/Coding/25-02-08-DDP.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,28 +108,20 @@ sidebar:
#### DDP Initialization

이제 이를 바탕으로 파이썬 코드 내에서 어떻게 세팅하는지 살펴보겠습니다. 각자마다 코딩하는 스타일이 있겠지만 저는 아래와 같이 작성하곤 합니다.
{% highlight python %}
args.device = 'cuda:0'
args.world_size = 1
args.rank = 0
args.local_rank = int(os.environ.get("LOCAL_RANK", 0))
torch.cuda.set_device(args.local_rank)
torch.distributed.init_process_group(backend='nccl', init_method='env://')
args.world_size = torch.distributed.get_world_size()
args.local_rank = torch.distributed.get_rank()
{% endhighlight %}

<figure class='highlight'>
<pre class='language-python' data-lang="python">
args.device = 'cuda:0'
args.world_size = 1
args.rank = 0
args.local_rank = int(os.environ.get("LOCAL_RANK", 0))
args.device = <span class='s'>'cuda:0'</span>
args.world_size = <span class='mi'>1</span>
args.rank = <span class='mi'>0</span>
args.local_rank = <span class='nb'>int</span>(os.environ.get(<span class='s'>"LOCAL_RANK"</span>, 0))
torch.cuda.set_device(args.local_rank)
torch.distributed.init_process_group(backend='nccl', init_method='env://')
torch.distributed.init_process_group(backend=<span class='s'>'nccl'</span>, init_method=<span class='s'>'env://'</span>)
args.world_size = torch.distributed.get_world_size()
args.local_rank = torch.distributed.get_rank()
</pre>
</figure>

---


Expand All @@ -149,18 +141,19 @@ args.local_rank = torch.distributed.get_rank()

#### Model에 DDP 적용하기

다음으로는 model에 DDP를 어떻게 적용 및 학습을 하는지 살펴보겠습니다.. (매우 간단합니다!!)
다음으로는 model에 DDP를 어떻게 적용 및 학습을 하는지 살펴보겠습니다.. (매우 간단합니다!!

<figure class='highlight'>
<pre class='language-python' data-lang="python">
from torch.nn.parallel import DistributedDataParallel as DDP\\
model = DDP(model,device_ids=[args.local_rank])\\
from <span class='nn'>torch.nn.parallel</span> import DistributedDataParallel as DDP
model = DDP(model,device_ids=[args.local_rank])
...
logits = model(x)
loss = loss_fn(logits, labels)
loss.backward()
</pre>
</figure>

---


Expand Down

0 comments on commit 2cbc546

Please sign in to comment.