Managing TLS certificates has never been easier. Not that long ago, running secure websites was a tedious job for engineers as they had to deal with complex business processes and chores. Who does not remember the times when you had to make a purchase requisition, get in touch with your vendor with your Certificate Signing Request (CSR), watch out for an email to validate your domain, and eventually announce a maintenance window that you will have fun with deploying the certificate in production?
Phew — I’m certainly glad those days are gone!
Shaken by the revolutionary non-profit Certificate Authority, Let’s Encrypt, and its ACME protocol, the market gradually moved into fully-automated solutions that enabled developers to deliver secure websites at no costs with the least effort.
Since day one, Traefik Proxy provides a native Let’s Encrypt integration to automate the full lifecycle of certificates. Without the need to handle any third-party tooling, Traefik Proxy is the natural choice for automated certificate management.
While using a single instance of Traefik Proxy with Let's Encrypt works like a charm, however, running multiple instances can raise some issues. If your production environment requires you to use Let's Encrypt with high availability (HA) in Kubernetes, you always have the option of Traefik Enterprise, which includes distributed Let's Encrypt as a supported feature.
But if you want to stick with Traefik Proxy, you have nothing to fear!
With Kubernetes we got a powerful and extensible platform to solve a lot of complex scenarios. cert-manager is a powerful solution that helps us automate and manage almost everything around TLS certificates. It provides a set of Custom Resource Definitions (CRD) for various scenarios and integrates well with native Ingress or Gateway resources.
cert-manager stores and caches certificates and private keys in Kubernetes secrets, making them highly available for further consumption by ingress controllers (like Traefik Proxy) or applications.
Note: By default, cert-manager does not clean up secrets automatically, allowing it to re-attach to already issued certificates and avoid issuing new certificates. This becomes very handy in scenarios when you need to create and delete lots of resources and would not like to be rate limited.
cert-manager can interact with a variety of sources to issue certificates including Let’s Encrypt, HashiCorp Vault as well as private PKI. For unsupported cases like AWS Private Certificate Authority, Google Cloud Certificate Authority Service or Cloudflare Origin CA the External Issuer allows you to extend cert-manager capabilities..
But enough talk! Time to get down to business and dig into how you can use cert-manager to extend Traefik Proxy’s capabilities.
Prerequisites
To follow this tutorial, you’ll need the following:
- A Kubernetes cluster >= v1.20
- A public hosted DNS domain for Let’s Encrypt — for the purpose of this article I will use Cloudflare
- A Kubernetes native ingress controller: Traefik Proxy 2.9, you can install the helm chart with this command:
helm install traefik traefik/traefik - cert-manager 1.10 which you can install with this command:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml - A service providing a web port. In this tutorial, I’ll be using whoami as an example:
apiVersion: v1kind: Namespacemetadata: name: whoami---apiVersion: v1kind: Servicemetadata: name: whoami namespace: whoamispec: ports: - name: web port: 80 targetPort: web selector: app: whoami---apiVersion: apps/v1kind: Deploymentmetadata: name: whoami namespace: whoamispec: selector: matchLabels: app: whoami template: metadata: labels: app: whoami spec: containers: - name: whoami image: traefik/whoami ports: - name: web containerPort: 80
Traefik Proxy with cert-manager and Let’s Encrypt
Let’s explore how we can secure a web application in combination with a Kubernetes ingress controller like Traefik Proxy and cert-manager. Let’s Encrypt provides multiple challenge types to validate control of a domain name. Depending on your requirements you may choose HTTP-01 when your service is public reachable or DNS-01 for private endpoints.
Please be aware of rate limits when using lets encrypt. To avoid unpleasant surprises it is recommended to use the Let’s Encrypt staging environment:
staging: https://acme-staging-v02.api.letsencrypt.org/directory`production: https://acme-v02.api.letsencrypt.org/directoryHTTP challenge
For most common scenarios the HTTP-01 challenge is a convenient start to solve an ACME based validation. To make this scenario work, Traefik Proxy needs to be reachable from the internet on HTTP port 80, and the used DNS domain has to be configured to point to it.
When a new certificate needs to be issued (or renewed), cert-manager will create a temporary Ingress resource to route requests made by the ACME server to the specific matched host and ./well-known/acme-challenge/xxx path, so it can answer with the desired response.
Implementing the challenge
First you need to define a new cert-manager Issuer to represent a certificate issuing authority. This example uses the ACME-based Certificate Authority in conjunction with Let’s Encrypt.
Note: You need to change the server to production to retrieve a certificate that will be accepted by your browser.
apiVersion: cert-manager.io/v1kind: Issuermetadata: name: le-example-http namespace: whoamispec: acme: email: [emailprotected] # We use the staging server here for testing to avoid hitting server: https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef: # if not existing, it will register a new account and stores it name: example-issuer-account-key solvers: - http01: # The ingressClass used to create the necessary ingress routes ingress: class: traefikNext, you’ll need a Kubernetes Ingress resource to define the domain for TLS we want to attach.
apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: whoami namespace: whoami annotations: cert-manager.io/issuer: "le-example-http"spec: tls: - hosts: - whoami.example.com secretName: tls-whoami-ingress-http rules: - host: whoami.example.com http: paths: - path: / pathType: Prefix backend: service: name: whoami port: name: webcert-manager automatically creates a new Certificate resource for the specified domain with the given secretName, provisions a CertificateRequest to request a signed certificate from one of the configured issuers, and stores the certificate and private key with the same name as the secret. The annotation cert-manager.io/issuer requires the name of the previously created Issuer and enables the resource to be managed by cert-manager.
Once the secret has been created, Traefik Proxy will fetch the certificate and private key and will serve it when the requested domain is called. Alternatively, you can also deploy a ClusterIssuer resource, which is accessible across all namespaces and referenced by the annotation cert-manager.io/cluster-issuer.
Note: cert-manager will not clean up certificates on its own, so they can be easily re-attached even if someone makes changes to the given Ingress object. If there is already an existing and valid certificate in place, it will be re-used.
DNS challenge
In some cases, you are not able to use the HTTP challenge (usually when your service is only internally available) and have to fall back to a DNS challenge. All you need to have in place is a registered DNS domain that can be resolved from the internet.
Unfortunately, cert-manager only supports a small range of DNS providers natively or dynamic DNS via RFC2136. Luckily there is the option to extend this with custom webhook solvers, so make sure to check out existing projects before implementing your own.
Implementing the challenge
The process looks almost the same as with the HTTP challenge. Instead of specifying the HTTP challenge, you need to set up the Issuer for using the DNS challenge. cert-manager will take care of creating the necessary validation records in the respected DNS zone.
apiVersion: v1kind: Secretmetadata: name: cloudflare-api-token-secrettype: OpaquestringData: api-token: <API Token>---apiVersion: cert-manager.io/v1kind: Issuermetadata: name: le-example-dns namespace: whoamispec: acme: email: [emailprotected] # We use the staging server here for testing to avoid hitting server: https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef: # if not existing, it will register a new account and stores it name: example-issuer-account-key solvers: - dns01: cloudflare: apiTokenSecretRef: name: cloudflare-api-token-secret key: api-token---apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: whoami namespace: whoami annotations: cert-manager.io/issuer: "le-example-dns"spec: tls: - hosts: - whoami.example.com secretName: tls-whoami-ingress-dns rules: - host: whoami.example.com http: paths: - path: / pathType: Prefix backend: service: name: whoami port: name: webTroubleshooting
All cert--manager resources provide handy status and event information. It helps you understand problems and verify everything is working as expected.
$ kubectl -n whoami get issuer -o wideNAME READY STATUSle-example-http True The ACME account was registered with the ACME server$ kubectl -n whoami get certificateRequest -o wideNAME APPROVED DENIED READY ISSUER STATUStls-whoami-ingress-http-fdw2x True True le-example-http Certificate fetched from issuer successfully$ kubectl -n whoami get certificatesNAME READY SECRET ISSUER STATUStls-whoami-ingress-http True tls-whoami-ingress-http le-example-http Certificate is up to date and has not expired$ kubectl -n whoami describe secret tls-whoami-ingress-httpAnnotations: cert-manager.io/alt-names: whoami.example.com cert-manager.io/certificate-name: tls-whoami-ingress-http cert-manager.io/common-name: whoami.example.com cert-manager.io/issuer-name: le-example-httpType: kubernetes.io/tlsData====tls.crt: 2449 bytestls.key: 1679 bytesSummary
This blog post just scratched the surface on the possibilities of cert-manager in conjunction with Let’s Encrypt. It already helps users to automate enrolling our application with publicly valid certificates for HTTPS while keeping it up to date.
Today, cert-manager is the almost perfect solution in Kubernetes for dealing with any kind of work with certificates. It is even possible to create your own simple private PKI without the need to deal with any CLI tools for automation.
I’d also recommend you explore more advanced features and use cases, like securing your pod-to-pod communication by leveraging the CSI driver for mTLS or the CSI SPIFFE driver.
Did you know that Traefik Proxy 3.0 Beta 1 added native support for SPIFFE? Check out the latest beta version of Traefik Proxy, play around with the new features and capabilities, and don’t forget to share your feedback!
