Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: added deletion overview with very high level chart. #288 #295

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs/started/concepts/deletion_overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
id: deletion-overview
title: Deletion Overview
---

Deletion in rucio can be performed in broadly two ways.

- Rule based deletion:

Each rule have a corresponding lifetime associated with it (default is None). The attribute `expires at` of the rule is time when the lifetime is set plus the lifetime associated with rule.
Also each rule can also have attribute called `lock`, if a rule is locked then the rule is protected from any deletion.
Daemon `Judge-cleaner` is responsible for handling expired rule. It checks all the expired rule and if the rule is locked no any action os taken else it removes the rule (delet rule) and set a tombstone to the replica associated with the rule.

- DID based deletion:

On the other hand, each dids also has lifetime associated with it (default is None). The metadata `expires at` of the did is calculated same a rule.
Daemon `Undertaker-undertaker` is responsible for handling expired dids. It checks all the expired dids and gets all the rules associated with the did. If rule of did is NOT locked then it removes all the rules of the did, set tombstone to all replicas , removes the did itself. If a did is has child , it removes all the child dids and sets tombstone for the child's replicas.

**The did expiration overrules the rule expiration. But the locked rules are protected.**

After the tombstone is set for replicas, actual file deletion is done by daemon `Reaper-repear`. It deletes the replicas RSE by RSE. This daemon check the `availability_delete` atrribute of RSEs and if True it then checks what mode of deletion is set in RSEs.
The deletion service supports two different modes: greedy and non-greedy.

- Greedy

The undertaker daemon gets all the replicas with tombstone in RSEs and immediately delete all replicas.

- Non-greedy

The undertaker daemon first check if the free space is needed in RSEs. The needed free space is the difference of minimum free space (attrribute set for RSE) and actual free space in RSE. If the needed free space is negative then no deletion occurs.
else it gets all the replicas with tombstone in RSEs. The iy useses a Least Recently Used (LRU) algorithm to select replicas for deletion.


![Judge-cleaner chart](/img/Judge-cleaner.png)

![Undertaker-undertaker chart](/img/Undertaker-undertaker.png)

![Repear-repear chart](/img/Repear-repear.png)

9 changes: 0 additions & 9 deletions docs/started/concepts/replica_management.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,6 @@ minimum set of necessary transfer primitives to satisfy all rules.
Notifications can be provided for rules and their underlying transfer
requests. All transfer requests are transient.

The deletion service supports two different modes: greedy and non-greedy. Greedy
means that the service tries to immediately delete all replicas which are not
protected by a replication rule. Non-greedy deletion is triggered when storage
policy dictates that space must be freed. The deletion service will look for
replicas on that RSE which can be deleted without violating any replication
rule. The deletion service will use a Least Recently Used (LRU) algorithm to
select replicas for deletion. The deletion service will also immediately delete
all replicas of any file which is declared obsolete.

Some examples of replication rules are listed
[here](replication_rules_examples.md).

Expand Down
Binary file added website/static/img/Judge-cleaner.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/Repear-repear.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/Undertaker-undertaker.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading