trust-manager
trust-manager is the easiest way to manage trust bundles in Kubernetes and OpenShift clusters.
It orchestrates bundles of trusted X.509 certificates which are primarily used for validating certificates during a TLS handshake but can be used in other situations, too.
Overview
trust-manager is a small Kubernetes operator which aims to help reduce the overhead of managing TLS trust bundles in your clusters.
It adds the Bundle
custom Kubernetes resource (CRD) which can read input from various sources
and combine the resultant certificates into a bundle ready to be used by your applications.
trust-manager ensures that it's both quick and easy to keep your trusted certificates up-to-date and enables cluster administrators to easily automate providing a secure bundle without having to worry about rebuilding containers to update trust stores.
It's designed to complement cert-manager and works well when consuming CA certificates from a
cert-manager Issuer
or ClusterIssuer
but can be used entirely independently of cert-manager
if needed.
Installation
See the installation guide for instructions on how to install trust-manager.
Usage
trust-manager is intentionally simple, adding just one new Kubernetes CustomResourceDefintion
: Bundle
.
A Bundle
represents a set of X.509 certificates that should be distributed across a cluster.
All Bundle
s are cluster scoped.
Bundle
s comprise a list of sources
from which trust-manager will assemble the final bundle, along with
a target
describing how and where the resulting bundle will be written.
An example Bundle
might look like this:
apiVersion: trust.cert-manager.io/v1alpha1kind: Bundlemetadata:name: my-org.com # The bundle name will also be used for the targetspec:sources:# Include a bundle of publicly trusted certificates which can be# used to validate most TLS certificates on the internet, such as# those issued by Let's Encrypt, Google, Amazon and others.- useDefaultCAs: true# A Secret in the "trust" namespace; see "Trust Namespace" below for further details- secret:name: "my-db-tls"key: "ca.crt"# Here is another Secret source, but this time using a label selector instead of a Secret's name.- secret:selector:matchLabels:fruit: applekey: "ca.crt"# And another Secret source, but this time instead of specifying a key from the Secret data,# including all certificates from every key- secret:name: "my-regional-cas"includeAllKeys: true# A ConfigMap in the "trust" namespace; see "Trust Namespace" below for further details- configMap:name: "my-org.net"key: "root-certs.pem"# Here is another ConfigMap source, but this time using a label selector instead of a ConfigMap's name.- configMap:selector:matchLabels:fruit: applekey: "ca.crt"# And another ConfigMap source, but this time instead of specifying a key from the ConfigMap data,# including all certificates from every key- secret:name: "my-org-cas"includeAllKeys: true# A manually specified string- inLine: |-----BEGIN CERTIFICATE-----MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl....0V3NCaQrXoh+3xrXgX/vMdijYLUSo/YPEWmo-----END CERTIFICATE-----target:# Sync the bundle to a ConfigMap called `my-org.com` in every namespace which# has the label "linkerd.io/inject=enabled"# All ConfigMaps will include a PEM-formatted bundle, here named "root-certs.pem"# and in this case we also request binary formatted bundles in JKS and PKCS#12 formats,# here named "bundle.jks" and "bundle.p12".configMap:key: "root-certs.pem"additionalFormats:jks:key: "bundle.jks"pkcs12:key: "bundle.p12"namespaceSelector:matchLabels:linkerd.io/inject: "enabled"
All sources and target options are documented in the trust-manager API reference documentation.
Sources
Bundle
resources currently support several source types:
configMap
- aConfigMap
resource in the trust-manager namespacesecret
- aSecret
resource in the trust-manager namespaceinLine
- a manually specified string containing at least one certificateuseDefaultCAs
- usually, a bundle of publicly trusted certificates
Both ConfigMap
and Secret
, support specifying a data key (key
) that contains at least one certificate or use the
includeAllKeys
option to include all certificates from every key in the resource. The latter is useful in dynamic
environments where key names are only known at runtime. When defining a ConfigMap
or Secret
source, the key
and
includeAllKeys
fields are mutually exclusive: only one must be set, but not both.
Both ConfigMap
and Secret
also support specifying label selectors to select multiple resources at once, which is useful in dynamic
environments where the name of the ConfigMap
or Secret
is known only at runtime. When adding a source, either of type ConfigMap
or Secret
,
the fields name
and selector
are mutually exclusive: one must be set, but not both.
These features can be combined to, for instance, select all Secrets with a specific label and include every certificate from each key within those Secrets.
Targets
ConfigMap
is the default target type, but as of v0.7.0 trust-manager also supports Secret
resources as targets.
Support for Secret
targets must be explicitly enabled in the trust-manager controller; see details below under "Enable Secret targets".
All Bundle
targets are written to ConfigMap
s (and/or Secret
s) whose name matches that of the
Bundle
, and every target has a PEM-formatted bundle included.
Users can also optionally choose to write JKS/PKCS#12 formatted binary trust store(s) to targets. JKS has been supported since v0.5.0, and PKCS#12 since v0.7.0.
Applications consuming JKS and PKCS#12 trust stores often require a password to be set for legacy reasons. These passwords are often security theater - either they use very weak encryption or the passwords are provided in plaintext next to the files they encrypt which defeats the purpose of having them.
Trust bundles do not contain private keys, and so for most use cases there wouldn't be any security benefit to encrypting them. As such, passwords for trust stores are set by default to changeit
for JKS and ""
(the empty string or "password-less") for PKCS#12.
Recent releases allow you to change that password by setting the bundle YAML file spec.target.additionalFormats.jks.password
and spec.target.additionalFormats.pkcs12.password
.
Older releases have the current default values hard-coded and they can not be changed. For more information read why password are not helpful.
Namespace Selector
A target's namespaceSelector
is used to restrict which Namespaces your Bundle
's target
should be synced to.
namespaceSelector
supports the field matchLabels
.
Please see Kubernetes documentation for more information about how label selectors can be configured.
If namespaceSelector
is empty, a Bundle
's target will be synced to all Namespaces.
â ī¸ A future update to trust-manager will change this behavior so that an empty namespace selector will sync only to the trust-manager namespace by default.
Quick Start Example
Let's get started with an example of creating our own Bundle
!
First we'll create a demo cluster:
git clone https://github.com/cert-manager/trust-manager trust-managercd trust-managermake demo
Once we have a running cluster, we can create a Bundle
using the default CAs which were configured
when trust-manager started up. Since we've installed trust-manager using Helm, our default CA package
contains publicly trusted certificates derived from a Debian container.
kubectl --kubeconfig ./bin/kubeconfig.yaml apply -f - <<EOFapiVersion: trust.cert-manager.io/v1alpha1kind: Bundlemetadata:name: example-bundlespec:sources:- useDefaultCAs: truetarget:configMap:key: "trust-bundle.pem"EOF
That was easy! Now let's check that everything synced OK and that our ConfigMap
has been written:
kubectl --kubeconfig ./bin/kubeconfig.yaml get bundle example-bundle | lesskubectl --kubeconfig ./bin/kubeconfig.yaml get configmap example-bundle -o "jsonpath={.data['trust-bundle\.pem']}" | less
Great - we've got a trust bundle. We could use that for our containers right away but let's go a little further and
create a dummy "organization CA" which we'll want to include in our Bundle
.
We'll generate our dummy organization certificate with cert-manager:
kubectl --kubeconfig ./bin/kubeconfig.yaml apply -f - <<EOFapiVersion: cert-manager.io/v1kind: ClusterIssuermetadata:name: trust-manager-selfsigned-issuerspec:selfSigned: {}---apiVersion: cert-manager.io/v1kind: Certificatemetadata:name: trust-manager-example-canamespace: cert-managerspec:isCA: truecommonName: trust-manager-example-casecretName: trust-manager-example-ca-secretprivateKey:algorithm: ECDSAsize: 256issuerRef:name: trust-manager-selfsigned-issuerkind: ClusterIssuergroup: cert-manager.ioEOF
Now let's check that Secret
that cert-manager created for us:
kubectl --kubeconfig ./bin/kubeconfig.yaml get -n cert-manager secret trust-manager-example-ca-secret -o"jsonpath={.data['tls\.crt']}" | base64 -d# tls.crt will contain a PEM certificate, starting with -----BEGIN CERTIFICATE-----
đ¤ Wondering why we used
tls.crt
and notca.crt
? More details below.
Finally, we'll update our Bundle
to include our new private CA:
kubectl --kubeconfig ./bin/kubeconfig.yaml apply -f - <<EOFapiVersion: trust.cert-manager.io/v1alpha1kind: Bundlemetadata:name: example-bundlespec:sources:- useDefaultCAs: true- secret:name: "trust-manager-example-ca-secret"key: "tls.crt"target:configMap:key: "trust-bundle.pem"EOF
And we're done! The example-bundle
ConfigMap
should already be updated.
If you inspect the ConfigMap
again, the last certificate you'll see in the list
should be our new dummy CA.
Securely Maintaining A trust-manager Installation
If you choose the useDefaultCAs
source on any of your Bundle
resources, it's important that you keep your
default CA package image up to date. Failing to do so would be the equivalent of failing to run apt-get upgrade ca-certificates
when installing a public trust bundle in a Debian container.
trust-manager has been designed in such a way that any version of any default CA package should work with any
version of trust-manager which supports default CAs (v0.4.0
and later). There should be no risk to the stability
of trust-manager from upgrading.
If you're using an official cert-manager-provided Debian CA package (which is the default), you should check which version you have and compare against the latest package version.
The version can be configured with the .defaultPackageImage.tag
value on the Helm chart, and the version
is also written to the status
field on any synced Bundle resource which uses the default CA package.
Upgrading A Default CA Package Using Helm
Say we want to do an in-place upgrade of our default CA package to tagged version XYZ
- without upgrading trust-manager.
We'll assume the Helm release is called "trust-manager" and that we've installed into the cert-manager
namespace.
â ī¸ This upgrade process assumes that it's the only thing running. If another user or process changes Helm values while you're doing this process, you might overwrite their work.
First, we'll dump our current Helm values, so we don't lose them:
helm get values -n cert-manager trust-manager -oyaml > values.yaml
Next, if defaultPackageImage.tag
is already set in values.yaml
, update it. Otherwise, add it.
You can find the available tags on quay.io
.
# values.yaml...defaultPackageImage:tag: XYZ
These versions of the default package image tags are derived directly from the version of the ca-certificates
package in Debian.
Finally, apply back the changes, being sure to manually specify the version of trust-manager which is installed, to avoid also updating the trust-manager controller at the same as the default CA package:
# Get the currently installed version. You could do this manually if you find that easier.TRUST_MANAGER_VER=$(helm list --filter "^trust-manager$" -n cert-manager -ojson | jq -r ".[0].app_version")# Check the version makes senseecho $TRUST_MANAGER_VER# Run the upgradehelm upgrade -f values.yaml -n cert-manager trust-manager jetstack/trust-manager --version $TRUST_MANAGER_VER
If an incorrect tag is used, your deployment will fail and you'll likely need to use helm rollback
to get back
to a working state.
Preparing for Production
TLS can be complicated and there are many ways to misuse TLS certificates.
Here are some potential gotchas to be aware of before running trust-manager in production.
If you're planning on running trust-manager in production and you're using more than just the default CA package, we strongly advise you to read and understand this section. It could save you from causing an outage later.
âšī¸ These gotchas aren't specific to trust-manager and you could run into any of them with any method of managing TLS trust!
Bundling Intermediates
If you've ever used a Let's Encrypt client such as Certbot you'll probably have
seen that it generates several certificate files, such as cert.pem
, chain.pem
, and fullchain.pem
.
These various files are provided to support various different applications, which might require the certificate
and the chain to be given separately. For most users and applications fullchain.pem
is the only correct choice.
Unfortunately the existence of these files has the unfortunate side effect of people sometimes assuming that cert.pem
is the correct choice even when fullchain.pem
would be correct. This means that the rest of the chain will not
be sent when the certificate is used.
Often, a quick fix that seems to work for this is that clients add the chain to their trust store, which will seem to fix certificate errors in the short term. It's easy for this kind of "fix" to end up being embedded somewhere as a solution which others can follow.
This "fix" is dangerous; it means that the intermediate cannot be safely rotated without all trust stores which contain it being updated first.
Intermediates in this case become de facto root certificates, which completely defeats the point of having intermediate certificates in the first place.
Avoid using intermediates in any trust store wherever possible unless you're absolutely certain they should be included. An example of where it might be OK would be cross signing, which is not likely to be required in the general case.
It would be better to copy just the root certificate to a new ConfigMap
and use that as a source rather than trusting
an intermediate.
cert-manager Integration: ca.crt
vs tls.crt
If you're pointing trust-manager at a Secret
containing a cert-manager-issued certificate, you'll see two relevant
fields: ca.crt
and tls.crt
. (We're ignoring tls.key
- trust-manager definitely doesn't need to access that)
That leads to an obvious question: between ca.crt
and tls.crt
, which should I use for trust-manager?
Unfortunately, it's impossible to say in the general case which field is correct to use, but we can provide guidelines.
tls.crt
will generally contain multiple certificates which may not all be issuers and some of which are likely to be
intermediate certificates. If that's the case, you shouldn't use tls.crt
as a source. (See "Bundling Intermediates" above for details.)
ca.crt
might then seem like the more generally correct choice but it's important to bear in mind that it can only ever
be populated on a best-effort basis. The contents of ca.crt
depend on the Issuer
being configured correctly, and some
issuer types may not ever be able to provide a useful or correct entry for this field.
As a rule, you should prefer to create Bundles
exclusively using root certificates (again, see above), and so you should
only use whichever field has a single root certificate in it. Consider reading below about why you might not want to
actually rely directly on cert-manager-issued certificates.
cert-manager Integration: Intentionally Copying CA Certificates
It's very strange in the Kubernetes world to suggest intentionally adding a step which seems to make automating infrastructure harder, but in the case of TLS trust stores it can be a wise choice.
Say you have a cert-manager Issuer
which has the root certificate you want to trust in ca.crt
. It's tempting to
use the Secret
directly and point at ca.crt
, but a best practice would be to copy that root into a separate ConfigMap
(or Secret
).
The reason is - as with many TLS gotchas - certificate rotation. If you rotate your issuer such that it's issued from a new root
certificate, trust-manager will see the Secret
be updated and automatically update your trust bundle to include the new root -
immediately distrusting the old root.
That means that if any services were still using a certificate issued by the old root, they'll be distrusted and will break.
Rotation requires that both root certificates are trusted simultaneously for a period, or else that all issued certificates are rotated either before or at the same time as the old root.
Known Issues
kubectl describe
The useDefaultCAs
option hits a corner case inside kubectl describe
and is rendered as Use Default C As: true
. This is
purely cosmetic.