Istio Practical Practice (12)-Istio Extends the Validity Period of Self-Issued Certificates

For historical reasons, Istio’s self-signed certificates are only valid for one year. If you choose to use Istio’s self-signed certificates, you’ll need to plan to replace your root certificates before they expire. Expired root certificates can cause unexpected cluster-wide outages.

We believe that replacing root certificates and keys every year is a security best practice, and we will introduce how to complete the rotation process of root certificates and keys in subsequent content.

To find out the remaining validity of your root certificate, please refer to the first step in the procedure below.

We provide the following procedure for completing root certificate updates. The Envoy process does not need to be restarted to load the new root certificate, so long links will not be affected. To understand Envoy’s warm restart process and its scope of impact, please refer to the related documentation and blog.

1. Scenario

If you are not currently using the mutual TLS feature and do not plan to use it in the future, no action is required. You can choose to upgrade to 1.0.8, 1.1.8 or later versions to avoid this problem in the future.

If you are not currently using mutual TLS, but plan to adopt it in the future, it is recommended to follow the steps below to update.

If you are currently using mutual TLS backed by a self-signed certificate, follow the steps below and check whether your current deployment is affected.

2. Root certificate migration process

Check if the root certificate has expired:

Download the script to a machine that contains kubectl and access the cluster.

$ wget https://raw.githubusercontent.com/istio/tools/master/bin/root-transition.sh
$ chmod +x root-transition.sh
$ ./root-transition.sh check
...
===YOU HAVE 30 DAYS BEFORE THE ROOT CERT EXPIRES!=====

Perform the remaining steps before the root certificate expires to prevent system downtime.

Perform the root certificate update process:

During the update process, Envoy will use a hot restart to load the new certificate. This may have some impact on traffic. You can read the documentation and blog about Envoy hot start for more details.

If your Pilot does not have an Envoy Sidecar, you should install a Sidecar for Pilot. This is because Pilot has issues validating new workload certificates using the old root certificate, causing a disconnect between Pilot and Evnoy. Please refer to the steps to learn how to check. The Istio upgrade guide installs Envoy Sidecar for Pilot by default.

$ ./root-transition.sh transition
Create new ca cert, with trust domain as cluster.local
Wed Jun 5 19:11:15 PDT 2019 delete old ca secret
secret "istio-ca-secret" deleted
Wed Jun 5 19:11:15 PDT 2019 create new ca secret
secret/istio-ca-secret created
pod "istio-citadel-8574b88bcd-j7v2d" deleted
Wed Jun 5 19:11:18 PDT 2019 restarted Citadel, checking status
NAME READY STATUS RESTARTS AGE
istio-citadel-8574b88bcd-l2g74 1/1 Running 0 3s
New root certificate:
Certificate:
    Data:
        ...
        Validity
            Not Before: Jun 6 03:24:43 2019 GMT
            Not After: Jun 3 03:24:43 2029 GMT
        Subject: O = cluster.local
        ...
Your old certificate is stored as old-ca-cert.pem, and your private key is stored as ca-key.pem
Please save them safely and privately.

Check the newly generated certificate for the workload:

$ ./root-transition.sh verify
...
Checking the current root CA certificate is propagated to all the Istio-managed workload secrets in the cluster.
Root cert MD5 is 8fa8229ab89122edba73706e49a55e4c
Checking namespace: default
  Secret default.istio.default is updated.
  Secret default.istio.sleep is updated.
Checking namespace: istio-system
  Secret istio-system.istio.default is updated.
  ...
------All Istio keys and certificates are updated in secret!

Citadel’s certificate distribution process takes a while to complete, so if the command fails, you can try again in a few minutes.

Upgrade to Istio 1.0.8, 1.1.8 or newer:

This step ensures that the control plane components and Envoy sidecar are all loaded with the new certificates and keys

Upgrade the control plane and istio-proxy to 1.0.8, 1.1.8, or newer. Please follow the instructions for the Istio update process to complete this step.
Check that Envoy has loaded the new workload certificate:

You can check whether Envoy has received the new certificate. To check for a Pod named foo in namespace bar, verify its Envoy certificate:
```
$ kubectl exec -it foo -c istio-proxy -n bar -- curl http://localhost:15000/certs | head -c 1000
{
 "certificates": [
  {
   "ca_cert": [
      ...
      "valid_from": "2019-06-06T03:24:43Z",
      "expiration_time": ...
   ],
   "cert_chain": [
    {
      ...
    }
```
See valid_from in ca_cert. If it matches the Not Before content of the new certificate in step 2, Envoy has loaded the new root certificate.

FAQ

Can I upgrade to Istio first? 1.0.8, 1.1.8 or newer versions, and then update the root certificate?

Yes. You can upgrade to Istio 1.0.8, 1.1.8 or newer versions as usual. After the upgrade, complete the root certificate update according to the content in step 4, and manually restart Galley, Pilot, and sidecar-injector to load the new root certificate.

Why is my workload not loading the new certificate (step 5)?

First make sure to upgrade istio-proxy to Istio 1.0.8, 1.1.8 or later in step 4.

If you are using Istio 1.1.3-1.1.7, Envoy may not perform a hot restart due to the generation of new certificates.

Why is my Pilot not working, and is printing “handshake error” in the log?

This may be because Pilot does not configure Envoy Sidecar when controlPlaneSecurity is enabled. In this case, Galley and Pilot need to be restarted to ensure they load the new certificate. For example, the following command redeploys Galley and Pilot by deleting the Pod:

<code>$ kubectl delete po <galley-pod> -n istio-system $ kubectl delete po <pilot-pod> -n istio-system </code>

How to check Pilot’s Sidecar

If the following command displays 1/1, it means that Pilot does not have a sidecar; if it displays 2/2, it means that Pilot has a sidecar.

<code>$ kubectl get po -l istio=pilot -n istio-system istio-pilot-569bc6d9c-tfwjr 1/1 Running 0 11m </code>

Cannot use sidecar-injector to deploy new workload

This may occur if you have not updated to Istio 1.0.8, 1.1.8 or later. Please try restarting the Sidecar injector. The certificate will be loaded after the restart:

$ kubectl delete po -l istio=sidecar-injector -n istio-system
pod "istio-sidecar-injector-788bd8fc48-x9gdc" deleted

The knowledge points of the article match the official knowledge files, and you can further learn related knowledge. Network Skill TreeHomepageOverview 42121 people are learning the system