Backup and Restore Resources

If you need to uninstall cert-manager, or transfer your installation to a new cluster, you can backup all of cert-manager's configuration in order to later re-install.

Backing up cert-manager resource configuration

The following commands will back up the configuration of cert-manager resources. Doing that might be useful before upgrading cert-manager. As this backup does not include the Secrets containing the X.509 certificates, restoring to a cluster that does not already have those Secret objects will result in the certificates being reissued.

Backup

To backup all of your cert-manager configuration resources, run:

kubectl get --all-namespaces -oyaml issuer,clusterissuer,cert > backup.yaml

If you are transferring data to a new cluster, you may also need to copy across additional Secret resources that are referenced by your configured Issuers, such as:

CA Issuers

• The root CA Secret referenced by issuer.spec.ca.secretName

Vault Issuers

• The token authentication Secret referenced by issuer.spec.vault.auth.tokenSecretRef
• The AppRole configuration Secret referenced by issuer.spec.vault.auth.appRole.secretRef

ACME Issuers

• The ACME account private key Secret referenced by issuer.acme.privateKeySecretRef
• Any Secrets referenced by DNS providers configured under the issuer.acme.dns01.providers and issuer.acme.solvers.dns01 fields.

Restore

In order to restore your configuration, you can kubectl apply the files created above after installing cert-manager, with the exception of the uid and resourceVersion fields that do not need to be restored:

kubectl apply -f <(awk '!/^ *(resourceVersion|uid): [^ ]+$/' backup.yaml)

Full cluster backup and restore

This section refers to backing up and restoring 'all' Kubernetes resources in a cluster — including some cert-manager ones — for scenarios such as disaster recovery, cluster migration etc.

Note: We have tested this process on simple Kubernetes test clusters with a limited set of Kubernetes releases. To avoid data loss, please test both the backup and the restore strategy on your own cluster before depending upon it in production. If you encounter any errors, please open a GitHub issue or a PR to document variations on this process for different Kubernetes environments.

Avoiding unnecessary certificate reissuance

Order of restore

If cert-manager does not find a Kubernetes Secret with an X.509 certificate for a Certificate, reissuance will be triggered. To avoid unnecessary reissuance after a restore, ensure that Secrets are restored before Certificates. Similarly, Secrets should be restored before Ingresses if you are using ingress-shim.

Excluding some cert-manager resources from backup

cert-manager has a number of custom resources that are designed to represent a point-in-time operation. An example would be a CertificateRequest that represents a one-time request for an X.509 certificate. The status of these resources can depend on other ephemeral resources (such as a temporary Secret holding a private key) so cert-manager might not be able to correctly recreate the state of these resources at a later point.

In most cases backup and restore tools will not restore the statuses of custom resources, so including such one-time resources in a backup can result in an unnecessary reissuance after a restore as without the status fields cert-manager will not be able to tell that, for example, an Order has already been fulfilled. To avoid unnecessary reissuance, we recommend that Orders and Challenges are excluded from the backup. We also don't recommend backing up CertificateRequests, see Backing up CertificateRequests

Restoring Ingress Certificates

A Certificate created for an Ingress via ingress-shim will have an owner reference pointing to the Ingress resource. cert-manager uses the owner reference to verify that the Certificate 'belongs' to that Ingress and will not attempt to create/correct it for an existing Certificate. After a full cluster recreation, a restored owner reference would probably be incorrect (Ingress UUID will have changed). The incorrect owner reference could lead to a situation where updates to the Ingress (i.e a new DNS name) are not applied to the Certificate.

To avoid this issue, in most cases Certificates created via ingress-shim should be excluded from the backup. Given that the restore happens in the correct order (Secret with the X.509 certificate restored before the Ingress) cert-manager will be able to create a new Certificate for the Ingress and determine that the existing Secret is for that Certificate.

Velero

We have tested backup and restore with velero v1.12.2 and cert-manager version v1.13.2.

A few potential edge cases:

• Ensure that the backups include cert-manager CRDs. For example, we have seen that if --exclude-namespaces flag is passed to velero backup create, CRDs for which there are no actual resources to be included in the backup might also not be included in backup unless --include-cluster-resources=true flag is also passed to the backup command.

• Velero does not restore statuses of custom resources, so you should probably exclude Orders, Challenges and CertificateRequests from the backup, see Excluding some cert-manager resources from backup.

• Velero's default restore order (Secrets before Ingresses, Custom Resources restored last), should ensure that there is no unnecessary certificate reissuance due to the order of restore operation, see Order of restore.

• When restoring the deployment of cert-manager itself, it may be necessary to restore cert-manager's RBAC resources before the rest of the deployment. This is because cert-manager's controller needs to be able to create Certificate's for the cert-manager's webhook before the webhook can become ready. In order to do this, the controller needs the right permissions. Since Velero by default restores pods before RBAC resources, the restore might get stuck waiting for the webhook pod to become ready.

• Velero does not restore owner references, so it may be necessary to exclude Certificates created for Ingresses from the backup even when not re-creating the Ingress itself. See Restoring Ingress Certificates.

Example backup and restore using Velero

The following command will create a backup of all Kubernetes resources in the default and cert-manager namespaces, excluding Orders, Challenges and CertificateRequests (see above):

velero backup create \
  full-backup \
  --include-namespaces cert-manager,default \
  --include-cluster-resources=true \
  --exclude-resources challenges.acme.cert-manager.io,orders.acme.cert-manager.io,certificaterequests.cert-manager.io

To workaround Velero not restoring owner references, you can restore the backup in two steps: first restore the Secrets and Ingresses and the cert-manager deployment, second restore the Certificate resources. This will allow cert-manager's controller to create the Certificate for the ingresses and set the owner reference. The second restore will then restore the manually created Certificates and detect that the generated Certificates for the Ingresses already exist and will not attempt to recreate them.

1. Restore everything except Certificate resources:

velero restore create \
  --from-backup full-backup \
  --exclude-resources certificates.cert-manager.io

2. Wait for cert-manager to create the Certificates for the Ingresses (if cert-manager is having RBAC/ webhook issues, you might have to manually restart the deployments). After the auto-generated Certificates are created, restore the manually created Certificates:

velero restore create \
  --from-backup full-backup

Backing up CertificateRequests

We no longer recommend including CertificateRequest resources in a backup for most scenarios. CertificateRequests are designed to represent a one-time request for an X.509 certificate. Once the request has been fulfilled, CertificateRequest can usually be safely deleted¹. In most cases (such as when a CertificateRequest has been created for a Certificate) a new CertificateRequest will be created when needed (i.e at a time of a renewal of a Certificate). In v1.3.0 , as part of our work towards policy implementation we introduced identity fields for CertificateRequest resources where, at a time of creation, cert-mananager's webhook updates CertificateRequest's spec with immutable identity fields, representing the identity of the creator of the CertificateRequest. This introduces some extra complexity for backing up and restoring CertificateRequests as the identity of the restorer might differ from that of the original creator and in most cases a restored CertificateRequest would likely end up with incorrect state.