Rancher 2.x and Let’s Encrypt, with cert-manager and nginx ingress

Introduction

I’ve written this for those who, like me, want to use Let’s Encrypt certificates in Rancher 2.x for their workloads.

With 2.x, Rancher Labs changed the way Rancher operates drastically.  Cattle are no more, instead it uses Kubernetes as the orchestration engine and has morphed into a wrapper around Kubernetes providing a nifty GUI for administration.

In this lesson, I will assume that you have a Kubernetes cluster already running configured with a Rancher 2.x server (or HA).  I will also assume you have a basic understanding of Rancher 2.x, Kubernetes and Let’s Encrypt.

The video version of this tutorial can be viewed at our YouTube Channel!

Kubectl

The first step is configuring your local kubectl to work with your setup.  In Rancher 2.x, this is accomplished by going to your cluster and clicking the “kubectl config” button.

Next click “Copy to Clipboard”, as illustrated here.

After that, paste the contents into ~/.kube/config on Linux or OS X, or %HOMEPATH%\.kube\config on Windows.

To test it is working, run kubectl cluster-info. If no errors are displayed in the output, then your kubectl is configured for this cluster.  If you need to configure multiple clusters, please consult the Kubernetes documentation for it https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/.

Loading Helm Stable into Catalog

The next step involves going into your Catalog Settings and enabling Helm Stable if you have not already done so.  Access your Rancher Server, next to the Rancher Logo is your cluster/project selection, click it and click “Global”.

Next click “Catalogs” in the menu bar.

Then under Helm Stable, make sure the “Enabled” is green.  If it is not, click it to enable it.  Your Helm Stable should look like this:

If you had just enabled it, give Rancher a few minutes to download the charts.  If you wish to check the progress, execute docker container logs (rancher/rancher container id) -f.  It does take several minutes to download and import the charts, so be patient.

Install cert-manager

After enabling Helm Stable, head over to the Catalog Apps in your project.  Cert-manager can be deployed to any namespace, but I prefer to deploy it to default.  Access your Catalog Apps first:

Click “Launch”:

Then search for cert and click the “View Details” button on the cert-manager chart.

There are no answers really needed for this chart at this stage, so set the name and namespace to the desired settings and click “Launch”.  Once the pod is running, we’re ready to move on.

Resource Types for cert-manager

Cert-manager adds 3 resource types that we’ll discuss in the upcoming sections.

  1. Issuer – this defines a certificate issuer that is single scoped (limited to a single namespace)
  2. ClusterIssuer – same as an Issuer, but available across the entire cluster rather than scoped to a namespace
  3. Certificate – defines a certificate that needs to be requested and issued, and the specifications and configuration

The first issuer

I’m going to configure a DNS01 issuer first using CloudFlare DNS as my DNS provider.  Other supported DNS Providers include, at the time of this writing: Google CloudDNS, Amazon Route53, and Akamai FastDNS.  You can access the list here including specifics on defining each provider.

The first thing you need to do for CloudFlare DNS, is place your API Key into a secret.  Replace the api-key data (aGFoYSBnb3RjaGEh) listed here with a base64 encoded version of the API Key you get from CloudFlare.  You can use https://www.base64encode.org to encode your API Key or, from the command line, type echo “API Key” |base64 to get a base64 encoded version of your key.

> cat cf-dns-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-dns
type: Opaque
data:
  api-key: aGFoYSBnb3RjaGEh

Now that that is out of the way, let’s define our first Issuer.  We’ll be doing the Staging environment first, you can find the ACME URL for v2 staging environment here.

> cat issuer.yaml
apiVersion: certmananger.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: default
spec:
  acme:
    server: https://acme-staging-v2.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging
    dns01:
      providers:
      - name: cf-dns
        cloudflare:
          email: [email protected]
          apiKeySecretRef:
            name: cloudflare-dns
            key: api-key

So let’s break down these line by line real quick so spread some understanding.

apiVersion: certmanager.k8s.io/v1alpha1

This line defines the api version of the YAML file.  Since cert-manager uses its own resources, the resources are namespaced into cert-manager via “certmanager.k8s.io” and the version of cert-manager’s API is v1alpha1.

kind: Issuer

Here we set the resource type to Issuer

metadata:
  name: letsencrypt-staging
  namespace: default

Here we set the name of the issuer and the namespace it will be created in.  It must be in the same namespace as the ingress controller or in a namespace the ingress controller can access secrets from.

spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging

Here we tell cert-manager we’re using Let’s Encrypt, defining the URL endpoint to use (https://acme-staging-v02.api.letsencrypt.org/directory), our contact email in case of things like revocations, data breaches, certificate expiration, etc. and a secret reference where to store our account’s private key.

dns01:
  providers:
  - name: cf-dns
    cloudflare:
      email: [email protected]
      apiKeySecretRef:
        name: cloudflare-dns
        key: api-key

Here we tell cert-manager how to handle dns01 validations with this issuer.  In this case, I’m telling it to use cloudflare with the name cf-dns (as Issuers can use multiple providers, even multiple DNS providers), with my account email [email protected] and the account API Key being stored in the secret cloudflare-dns, key “api-key”.

Now that our first issuer is written, we can create it in the cluster by running kubectl create -f issuer.yamlAssuming no error messages are displayed, then your issuer has been deployed into the cluster.  Run kubectl describe issuer letsencrypt-staging to check on its progress.  The last few lines of the output should look like this:

Status:
  Acme:
    Uri: https://acme-v02.api.letsencrypt.org/acme/acct/123456789
  Conditions:
    Last Transition Time: 2018-05-26T16:36:56Z
    Message: The ACME account was registered with the ACME server
    Reason: ACMEAccountRegistered
    Status: True
    Type: Ready
Events: <none>

If it does not, you’ll need to read the error message displayed to determine the best course of action.

To add http01 validation to your Issuer, under acme, simply add:

http01: {}

This should be at the same indentation as dns01 in the example above and has no configuration options in the Issuer.

ClusterIssuer

ClusterIssuers are defined the exact same as an Issuer with two minor changes.  Firstly, your “Kind” attribute on line 2 above must read “ClusterIssuer”, and “namespace” under “metadata” must be removed.

To describe a ClusterIssuer with kubectl, run kubectl describe clusterissuer (clusterissuer name).  Notice the “clusterissuer” vice “issuer” following “describe”.

Certificate

Certificate is where the real meat and potatoes of this whole process comes into play.  In this example, I am going to display two sets of YAMLs, one with hard coded common and DNS names and another demonstrating Let’s Encrypts new Wildcard certificates.

> cat certificate.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: idealcoders-com
  namespace: idealcoders
spec:
  secretName: idealcoders-com-tls
  issuerRef:
    name: letsencrypt-staging
  commonName: www.idealcoders.com
  dnsNames:
  - idealcoders.com
  - api.idealcoders.com
  - mail.idealcoders.com
  acme:
    config:
    - dns01:
        provider: cf-dns
      domains:
      - www.idealcoders.com
      - idealcoders.com
      - api.idealcoders.com
      - mail.idealcoders.com

In this example, I am telling cert-manager that I want to create a certificate resource named “idealcoders-com” (I name my certificate resources by the TLD with “.” replaced with “-” to keep things standard), and I want my certificate deployed to the namespace “idealcoders”.  The certification must be deployed to the same namespace where the ingress shims exist, or can access, as well as the same namespace the Issuer is in.

Under spec, I define the secret name the certificate will be deployed to.  I usually use the name under metadata, and append “-tls” to the end so I know it’s the certificate.  I also must tell cert-manager which Issuer Reference to utilize.  Note: if using a ClusterIssuer reference, you must also specify kind: ClusterIssuer under the name with the same indentation level.

I set the common name, which will be the “primary” fully qualified domain name (FQDN) this certificate is issued against as “www.idealcoders.com” with alias FQDNs of idealcoders.com, api.idealcoders.com and mail.idealcoders.com.

Under acme, I define the configuration for Let’s Encrypt to know how to validate the FQDNs to ensure we have permission to have certificates issued.  The “cf-dns” next to provider is the one you defined in the issuer.yaml file (I used cf-dns above for my cloudflare definition).

Now let’s tweak this for HTTP01 validations.

> cat certificate-http01.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: idealcoders-com
  namespace: idealcoders
spec:
  secretName: idealcoders-com-tls
  issuerRef:
    name: letsencrypt-staging
  commonName: www.idealcoders.com
  dnsNames:
  - idealcoders.com
  - api.idealcoders.com
  - mail.idealcoders.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - www.idealcoders.com
      - idealcoders.com
      - api.idealcoders.com
      - mail.idealcoders.com

The big difference here is dns01 is replaced with http01, and instead of provider, you’re defining the ingressClass to use.  cert-manager uses nginx, and will define an nginx shim to handle the http01 validation requests for you.

You’ll also notice the list of FQDNs under “domains”.  Using this list, you can actually split up how names are validated, with some being a “dns01” request, maybe others on a different provider for dns01, and maybe some others under http01.  The only requirement is all names you want a certificate issued for must be under dnsNames and commonName, and each must appear once in the acme config section of the YAML.

Now for the Wildcard Certification, which can only use dns01 validation methods.

> cat certificate-wildcard.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: idealcoders-com
  namespace: idealcoders
spec:
  secretName: idealcoders-com-tls
  issuerRef:
    name: letsencrypt-staging
  commonName: '*.idealcoders.com'
  dnsNames:
  - idealcoders.com
  acme:
    config:
    - dns01:
        provider: cf-dns
      domains:
      - *.idealcoders.com
      - idealcoders.com

The common name in this case is set to the wildcard.  Because * is a special character, you will need to place quotations around the value.

And that’s it, deploy it into your cluster the same way you did the Issuer.  kubectl create -f certificate.yaml.

Eventually your certificates should issue and under “Status” of the describe you should see:

Status:
  Acme:
    Order:
      URL: https://acme-v02.api.letsencrypt.org/acme/order/12345678/12345678
  Conditions:
    Last Transition Time: 2018-06-04T18:25:26Z
    Message: Certificate issued successfully
    Reason: CertIssued
    Status: True
    Type: Ready

When you do, your certification is ready to go.  BUT! we’ve been doing the staging environment… so we’ve only been testing.  To change to production, it’s pretty simple.

Converting from staging to production

In your issuer, the server for production is the same URL except you remove the “-staging” portion of the url.  This is what my production YAML looks like.

> cat issuer-production.yaml
apiVersion: certmananger.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt
  namespace: default
spec:
  acme:
    server: https://acme-v2.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt
    dns01:
      providers:
      - name: cf-dns
        cloudflare:
          email: [email protected]
          apiKeySecretRef:
            name: cloudflare-dns
            key: api-key

Simple enough, 3 edits.  I removed “-staging” from metadata.name, spec.acme.privateKeySecretRef.name and from spec.acme.server.  The 3 lines I modified are bolded above.

Now you need to tweak your certificate file to use the production issuer.  First, if you created the staging certificate, DELETE the resource first.  kubectl delete -f certificate.yaml.  After that, go to Rancher, into your project, then “Resources” > “Certificates”.  Remove the certificate named “idealcoders-com-tls” as I named above (except the name you specified).

> cat certificate-production.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: idealcoders-com
  namespace: idealcoders
spec:
  secretName: idealcoders-com-tls
  issuerRef:
    name: letsencrypt
  commonName: www.idealcoders.com
  dnsNames:
  - idealcoders.com
  - api.idealcoders.com
  - mail.idealcoders.com
  acme:
    config:
    - dns01:
        provider: cf-dns
      domains:
      - www.idealcoders.com
      - idealcoders.com
      - api.idealcoders.com
      - mail.idealcoders.com

In the certificate YAML, there is only 1 line to change… and that’s the reference to the Issuer.  Now kubectl create -f both files, describe the certificate until it deploys and you’ll be ready to configure your Ingress shims.

Rancher 2.x Ingress Configuration

Create an Ingress like you normally would through Rancher, but then under “SSL/TLS Configuration”, you’ll need to select the Certificate and enter the host names that that certificate is valid for.  Here’s my example showing a wildcard certificate.  You can use “Add Hosts” to add a new host.  If you have multiple certificates that you need to use, use the “Add Certificate” button.

Questions?

If you have any questions, please do not hesitate to leave them as a comment here or ask at the Rancher Forums or Rancher Slack.

Video Reference

This tutorial was also created as a video to help new users and can be viewed at our YouTube channel!