Restic Database Example
Restic backup
Restic is a fast and secure backup program. The following example will use Restic to create a backup of a source volume.
A MySQL database will be used as the example application.
Creating source PVC to be backed up
Create a namespace called source
$ kubectl create ns source
$ kubectl annotate namespace source volsync.backube/privileged-movers="true"
Note
The second command to annotate the namespace is used to enable the restic data mover to run in privileged mode.
This is because this simple example runs MySQL as root. For your own applications, you can run unprivileged by
setting the moverSecurityContext
in your ReplicationSource/ReplicationDestination to match that of your
application in which case the namespace annotation will not be required. See the
permission model documentation for more details.
Deploy the source MySQL database.
$ kubectl -n source create -f examples/source-database/
Verify the database is running:
$ kubectl -n source get pods,pvc,volumesnapshots
NAME READY STATUS RESTARTS AGE
pod/mysql-87f849f8c-n9j7j 1/1 Running 1 58m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/mysql-pv-claim Bound pvc-adbf57f1-6399-4738-87c9-4c660d982a0f 2Gi RWO csi-hostpath-sc 60m
Add a new database:
$ kubectl exec --stdin --tty -n source $(kubectl get pods -n source | grep mysql | awk '{print $1}') -- /bin/bash
$ mysql -u root -p$MYSQL_ROOT_PASSWORD
> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
> create database synced;
> exit
$ exit
Restic Repository Setup
For the purpose of this tutorial we are using minio as the object storage target for the backup.
Start minio
:
$ hack/run-minio.sh
The restic-config
Secret configures the Restic repository parameters:
---
apiVersion: v1
kind: Secret
metadata:
name: restic-config
type: Opaque
stringData:
# The repository url
RESTIC_REPOSITORY: s3:http://minio.minio.svc.cluster.local:9000/restic-repo
# The repository encryption key
RESTIC_PASSWORD: my-secure-restic-password
# ENV vars specific to the back end
# https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html
AWS_ACCESS_KEY_ID: access
AWS_SECRET_ACCESS_KEY: password
The above will backup to a bucket called restic-repo
. If the same bucket will
be used for different PVCs or different uses, then make sure to specify a unique
path under restic-repo
. See Shared S3 Bucket Notes for more detail.
ReplicationSource
Start by configuring the source; a minimal example is shown below
---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: database-source
namespace: source
spec:
sourcePVC: mysql-pv-claim
trigger:
schedule: "*/30 * * * *"
restic:
pruneIntervalDays: 15
repository: restic-config
retain:
hourly: 1
daily: 1
weekly: 1
monthly: 1
yearly: 1
copyMethod: Clone
In the above ReplicationSource
object,
The PiT copy of the source data
mysql-pv-claim
will be created by cloning the source volume.The synchronization schedule,
.spec.trigger.schedule
, is defined by a cronspec, making the schedule very flexible. In this case, it will take a backup every 30 minutes.The restic repository configuration is provided via the
restic-config
Secret.pruneIntervalDays
defines the interval between Restic prune operations.The
retain
settings determine how many backups should be saved in the repository. Read more about restic forget.
Now, deploy the restic-config
followed by ReplicationSource
configuration.
$ kubectl create -f examples/restic/source-restic/source-restic.yaml -n source
$ kubectl create -f examples/restic/volsync_v1alpha1_replicationsource.yaml -n source
To verify the replication has completed, view the the ReplicationSource
.status
field.
$ kubectl -n source get ReplicationSource/database-source -oyaml
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: database-source
namespace: source
spec:
# ... lines omitted ...
status:
conditions:
- lastTransitionTime: "2021-05-17T18:16:35Z"
message: Reconcile complete
reason: ReconcileComplete
status: "True"
type: Reconciled
lastSyncDuration: 3m10.261673933s
lastSyncTime: "2021-05-17T18:19:45Z"
nextSyncTime: "2021-05-17T18:30:00Z"
restic: {}
In the above output, the lastSyncTime
shows the time when the last backup
completed.
The backup created by VolSync can be seen by directly accessing the Restic repository:
# In one window, create a port forward to access the minio server
$ kubectl port-forward --namespace minio svc/minio 9000:9000
# An another, access the repository w/ restic via the above forward
$ AWS_ACCESS_KEY_ID=access AWS_SECRET_ACCESS_KEY=password restic -r s3:http://127.0.0.1:9000/restic-repo snapshots
enter password for repository:
repository 03fd0c91 opened successfully, password is correct
created new cache in /home/jstrunk/.cache/restic
ID Time Host Tags Paths
------------------------------------------------------------
caebaa8e 2021-05-17 14:19:42 volsync /data
------------------------------------------------------------
1 snapshots
There is a snapshot in the restic repository created by the restic data mover.
Restoring the backup
To restore from the backup, create a destination, deploy restic-config
and
ReplicationDestination
on the destination.
$ kubectl create ns dest
$ kubectl annotate namespace dest volsync.backube/privileged-movers="true"
$ kubectl -n dest create -f examples/restic/source-restic/
To start the restore, create a empty PVC for the data:
$ kubectl -n dest create -f examples/source-database/mysql-pvc.yaml
persistentvolumeclaim/mysql-pv-claim created
Create the ReplicationDestination in the dest
namespace to restore the data:
---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
name: database-destination
spec:
trigger:
manual: restore
restic:
destinationPVC: mysql-pv-claim
repository: restic-config
copyMethod: Direct
$ kubectl -n dest create -f examples/restic/volsync_v1alpha1_replicationdestination.yaml
Once the restore is complete, the .status.lastManualSync
field will match
.spec.trigger.manual
.
To verify restore, deploy the MySQL database to the dest
namespace which will use the data that has
been restored from sourcePVC backup.
Create the Deployment, Service, and Secret.
$ kubectl create -n dest -f examples/destination-database/mysql-secret.yaml
$ kubectl create -n dest -f examples/destination-database/mysql-deployment.yaml
$ kubectl create -n dest -f examples/destination-database/mysql-service.yaml
Validate that the mysql pod is running within the environment.
$ kubectl get pods -n dest
NAME READY STATUS RESTARTS AGE
mysql-8b9c5c8d8-v6tg6 1/1 Running 0 38m
Connect to the mysql pod and list the databases to verify the synced database exists.
$ kubectl exec --stdin --tty -n dest $(kubectl get pods -n dest | grep mysql | awk '{print $1}') -- /bin/bash
$ mysql -u root -p$MYSQL_ROOT_PASSWORD
> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| synced |
| sys |
+--------------------+
5 rows in set (0.00 sec)
> exit
$ exit
Backing up multiple PVCs to the same S3 bucket
If using the same S3 bucket for multiple backups, then be aware of the following:
Each PVC to be backed up will need its own separate
restic-config
secret.Each
restic-config
secret may use the same s3 bucket name in the RESTIC_REPOSITORY, but they must each have a unique path underneath.
Example of backing up 2 PVCs, pvc-a
and pvc-b
:
A restic-config
and replicationsource
needs to be created for each pvc and each replicationsource
must refer to the correct restic-config
.
For pvc-a
:
---
# Restic-config Secret for pvc-a
apiVersion: v1
kind: Secret
metadata:
name: restic-config-a
type: Opaque
stringData:
# The repository url with pvc-a-backup as the subpath under the restic-repo bucket
RESTIC_REPOSITORY: s3:http://minio.minio.svc.cluster.local:9000/restic-repo/pvc-a-backup
# The repository encryption key
RESTIC_PASSWORD: my-secure-restic-password-pvc-a
# ENV vars specific to the back end
# https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html
AWS_ACCESS_KEY_ID: access
AWS_SECRET_ACCESS_KEY: password
---
# ReplicationSource for pvc-a
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: replication-source-pvc-a
namespace: source
spec:
sourcePVC: pvc-a
trigger:
schedule: "*/30 * * * *"
restic:
pruneIntervalDays: 15
repository: restic-config-a
retain:
hourly: 1
daily: 1
weekly: 1
monthly: 1
yearly: 1
copyMethod: Clone
For pvc-b
:
---
# Restic-config Secret for pvc-b
apiVersion: v1
kind: Secret
metadata:
name: restic-config-b
type: Opaque
stringData:
# The repository url with pvc-b-backup as the subpath under the restic-repo bucket
RESTIC_REPOSITORY: s3:http://minio.minio.svc.cluster.local:9000/restic-repo/pvc-b-backup
# The repository encryption key - using a different key from pvc-a. This will not prevent overwrites
# or deletes of the data for others who have access to the bucket, but will prevent reads/writes
# to the restic data in the pvc-b-backup folder for those without this encryption key.
RESTIC_PASSWORD: my-secure-restic-password-pvc-b
# ENV vars specific to the back end
# https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html
AWS_ACCESS_KEY_ID: access
AWS_SECRET_ACCESS_KEY: password
---
# ReplicationSource for pvc-b
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: replication-source-pvc-b
namespace: source
spec:
sourcePVC: pvc-b
trigger:
schedule: "*/30 * * * *"
restic:
pruneIntervalDays: 15
repository: restic-config-b
retain:
hourly: 1
daily: 1
weekly: 1
monthly: 1
yearly: 1
copyMethod: Clone