Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use vtbackup in scheduled backups #1946

Open
wants to merge 1 commit into
base: prod
Choose a base branch
from
Open

Use vtbackup in scheduled backups #1946

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions content/en/docs/22.0/get-started/operator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,19 @@ You can check the state of your cluster with `kubectl get pods`. After a few min

```bash
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
example-commerce-x-x-zone1-vtorc-c13ef6ff-5db4c77865-l96xq 1/1 Running 2 (2m49s ago) 5m16s
example-etcd-faf13de3-1 1/1 Running 0 5m17s
example-etcd-faf13de3-2 1/1 Running 0 5m17s
example-etcd-faf13de3-3 1/1 Running 0 5m17s
example-vttablet-zone1-2469782763-bfadd780 3/3 Running 1 (2m43s ago) 5m16s
example-vttablet-zone1-2548885007-46a852d0 3/3 Running 1 (2m47s ago) 5m16s
example-zone1-vtadmin-c03d7eae-7c6f6c98f8-f4f5z 2/2 Running 0 5m17s
example-zone1-vtctld-1d4dcad0-57b9d7bc4b-2tnqd 1/1 Running 2 (2m53s ago) 5m17s
example-zone1-vtgate-bc6cde92-7d445d676-x6npk 1/1 Running 2 (3m ago) 5m17s
vitess-operator-5f47c6c45d-bgqp2 1/1 Running 0 6m52s
NAME READY STATUS RESTARTS AGE
example-90089e05-vitessbackupstorage-subcontroller 1/1 Running 0 3m22s
example-commerce-x-x-vtbackup-init-c6db73c9 0/1 Completed 0 2m9s
example-commerce-x-x-zone1-vtorc-c13ef6ff-776f5676cb-7rcb7 1/1 Running 0 2m9s
example-etcd-faf13de3-1 1/1 Running 0 3m22s
example-etcd-faf13de3-2 1/1 Running 0 3m22s
example-etcd-faf13de3-3 1/1 Running 0 3m22s
example-vttablet-zone1-2469782763-bfadd780 3/3 Running 0 2m9s
example-vttablet-zone1-2548885007-46a852d0 3/3 Running 0 2m9s
example-zone1-vtadmin-c03d7eae-6db4c646bc-kslw9 2/2 Running 0 3m22s
example-zone1-vtctld-1d4dcad0-5674cc8448-qqjv7 1/1 Running 2 (3m7s ago) 3m22s
example-zone1-vtgate-bc6cde92-5bfb8f645-9flp9 1/1 Running 3 (2m47s ago) 3m22s
vitess-operator-5f4fb4dffb-7kpmq 1/1 Running 0 9m11s
```

## Setup Port-forward
Expand Down
28 changes: 16 additions & 12 deletions content/en/docs/22.0/user-guides/migration/move-tables.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,18 +98,22 @@ After a few minutes the pods should appear running:

```bash
$ kubectl get pods
example-commerce-x-x-zone1-vtorc-c13ef6ff-5d658d78d8-dvmnn 1/1 Running 1 (4m39s ago) 65d
example-etcd-faf13de3-1 1/1 Running 1 (4m39s ago) 65d
example-etcd-faf13de3-2 1/1 Running 1 (4m39s ago) 65d
example-etcd-faf13de3-3 1/1 Running 1 (4m39s ago) 65d
example-vttablet-zone1-1250593518-17c58396 3/3 Running 1 (27s ago) 32s
example-vttablet-zone1-2469782763-bfadd780 3/3 Running 3 (4m39s ago) 65d
example-vttablet-zone1-2548885007-46a852d0 3/3 Running 3 (4m39s ago) 65d
example-vttablet-zone1-3778123133-6f4ed5fc 3/3 Running 1 (26s ago) 32s
example-zone1-vtadmin-c03d7eae-7dcd4d75c7-szbwv 2/2 Running 2 (4m39s ago) 65d
example-zone1-vtctld-1d4dcad0-6b9cd54f8f-jmdt9 1/1 Running 2 (4m39s ago) 65d
example-zone1-vtgate-bc6cde92-856d44984b-lqfvg 1/1 Running 2 (4m6s ago) 65d
vitess-operator-8df7cc66b-6vtk6 1/1 Running 0 55s
NAME READY STATUS RESTARTS AGE
example-90089e05-vitessbackupstorage-subcontroller 1/1 Running 0 8m45s
example-commerce-x-x-zone1-vtorc-c13ef6ff-776f5676cb-7rcb7 1/1 Running 0 7m32s
example-customer-x-x-vtbackup-init-115abf93 0/1 Completed 0 112s
example-customer-x-x-zone1-vtorc-53d270f6-7df84448fb-hnnps 1/1 Running 0 112s
example-etcd-faf13de3-1 1/1 Running 0 8m45s
example-etcd-faf13de3-2 1/1 Running 0 8m45s
example-etcd-faf13de3-3 1/1 Running 0 8m45s
example-vttablet-zone1-1250593518-17c58396 3/3 Running 0 112s
example-vttablet-zone1-2469782763-bfadd780 3/3 Running 0 7m32s
example-vttablet-zone1-2548885007-46a852d0 3/3 Running 0 7m32s
example-vttablet-zone1-3778123133-6f4ed5fc 3/3 Running 0 112s
example-zone1-vtadmin-c03d7eae-6db4c646bc-kslw9 2/2 Running 0 8m45s
example-zone1-vtctld-1d4dcad0-5674cc8448-qqjv7 1/1 Running 2 (8m30s ago) 8m45s
example-zone1-vtgate-bc6cde92-5bfb8f645-9flp9 1/1 Running 3 (8m10s ago) 8m45s
vitess-operator-5f4fb4dffb-7kpmq 1/1 Running 0 14m
```

Again, the operator will promote one of the tablets to `PRIMARY` implicitly for you.
Expand Down
70 changes: 22 additions & 48 deletions .../docs/22.0/user-guides/operating-vitess/backup-and-restore/scheduled-backups.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ If you are not already familiar with [how backups work](../overview/) in Vitess
### Adding the schedule

{{< warning >}}
Please note that is not recommended to run production backups every minute. These schedules are only an example.
Please note that is not recommended to run production backups every two minutes. These schedules are only an example.
{{</ warning >}}

For this example we are going to create two schedules: each will be executed every minute, the first one will backup
For this example we are going to create two schedules: each will be executed every two minutes, the first one will backup
the two `customer` shards, and the second one will backup the `commerce` keyspace.
The backups will be stored directly inside the Minikube node, but if you want to backup to a cloud storage provider like S3, you can
change the `location` of the backup in `401_scheduled_backups.yaml`.
Expand All @@ -34,66 +34,40 @@ change the `location` of the backup in `401_scheduled_backups.yaml`.
kubectl apply -f 401_scheduled_backups.yaml
```

After a minute or so, we should see two new pods that were created by the operator. Under the hood, these pods
After two minutes, we should see three new pods that were created by the operator. Under the hood, these pods
are managed by a Kubernetes Job, and their goal is to take a backup of Vitess, as we defined in the `strategies` field
of the `401_scheduled_backups.yaml` file.

```bash
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
NAME READY STATUS RESTARTS AGE
...
example-vbsc-every-minute-commerce-ac6ff735-1715963880-4rt6r 0/1 Completed 0 31s
example-vbsc-every-minute-customer-8aaaa771-1715963880-n7cm7 0/1 Completed 0 31s
example-vbsc-commerce-ca641fc1-commerce-x-x-1740521040-b892727l 0/1 Completed 0 2m7s
example-vbsc-customer-05e172de-customer-80-x-1740521---04ar59dk 0/1 Completed 0 2m17s
example-vbsc-customer-05e172de-customer-x-80-1740521---6535gj8c 0/1 Completed 0 2m17s
...
```

### Listing backups

Now we can list the available backups, by getting the `vtb` (`VitessBackup`) objects in our Kubernetes cluster.
We can see we have three backups, that is because the schedule `every-minute-customer` takes two backups (one for each shard, `-80` and `80-`),
and the other schedule (`every-minute-commerce`) takes only one backup (for the shard `-`).

```bash
$ kubectl get vtb --no-headers
example-commerce-x-x-20240517-163802-97ece60f-8f2a3d47 111s
example-customer-80-x-20240517-163812-70e40ad-78a0d60b 58s
example-customer-x-80-20240517-163803-729301e-02e8899b 111s
```

Since we are running on Minikube, our backups are stored locally on the Minikube node, we can take a look at them like so:
Now we can list the available backups by getting the `vtb` (`VitessBackup`) objects in our Kubernetes cluster.
We can see six backups: For every keyspace/shard, one backup by the initial vtbackup pod, that spawns
as we create a new shard, and we can see a second backup created more recently (~46 seconds ago) by the scheduled
backup CRD.

```bash
$ minikube ssh
docker@minikube:~$ cd /tmp/example/
docker@minikube:/tmp/example$ ls
commerce customer
docker@minikube:/tmp/example$ ls commerce/-/ customer/
-80/ 80-/
docker@minikube:/tmp/example$ ls -l commerce/-/ customer/-80/ customer/80-/
commerce/-/:
total 0
drwxr-xr-x 2 999 docker 220 May 17 16:38 2024-05-17.163802.zone1-2548885007
drwxr-xr-x 2 999 docker 220 May 17 16:39 2024-05-17.163903.zone1-2548885007
drwxr-xr-x 2 999 docker 220 May 17 16:40 2024-05-17.164003.zone1-2548885007
drwxr-xr-x 2 999 docker 220 May 17 16:41 2024-05-17.164102.zone1-2548885007
drwxr-xr-x 2 999 docker 220 May 17 16:42 2024-05-17.164202.zone1-2548885007

customer/-80/:
total 0
drwxr-xr-x 2 999 docker 220 May 17 16:38 2024-05-17.163803.zone1-0120139806
drwxr-xr-x 2 999 docker 220 May 17 16:39 2024-05-17.163902.zone1-2289928654
drwxr-xr-x 2 999 docker 220 May 17 16:40 2024-05-17.164003.zone1-2289928654
drwxr-xr-x 2 999 docker 220 May 17 16:41 2024-05-17.164102.zone1-2289928654
drwxr-xr-x 2 999 docker 220 May 17 16:42 2024-05-17.164202.zone1-2289928654

customer/80-/:
total 0
drwxr-xr-x 2 999 docker 220 May 17 16:38 2024-05-17.163812.zone1-0118374573
drwxr-xr-x 2 999 docker 220 May 17 16:39 2024-05-17.163911.zone1-4277914223
drwxr-xr-x 2 999 docker 220 May 17 16:40 2024-05-17.164010.zone1-4277914223
drwxr-xr-x 2 999 docker 220 May 17 16:41 2024-05-17.164111.zone1-4277914223
drwxr-xr-x 2 999 docker 220 May 17 16:42 2024-05-17.164208.zone1-4277914223
docker@minikube:/tmp/example$
docker@minikube:~$ sudo chmod o+rwx -R /tmp/example/ # required to give Minikube permissions to read the Vitess backups
docker@minikube:~$ exit

$ kubectl get vtb
NAME AGE
example-commerce-x-x-20250225-214735-b82e162-74db1145 16m
example-commerce-x-x-20250225-220602-47f20b54-75d9f5c5 46s
example-customer-80-x-20250225-215751-5e335eca-fb525825 6m51s
example-customer-80-x-20250225-220602-706d77e3-b930b08c 46s
example-customer-x-80-20250225-215747-c74a3a36-45b61315 6m51s
example-customer-x-80-20250225-220558-26a8bbfe-0b58cca8 46s
```

### Cleanup
Expand Down