27. 手动轮换控制平面 TLS 凭证

Linkerd自动 mTLS 功能 使用一组 TLS 凭据为代理生成 TLS 证书信任锚颁发者证书私钥。 信任锚的有效期有限:365 天(如果由 linkerd install 生成)或 自定义值(如果手动生成)。

因此,对于预计会超过此生命周期的集群,您必须手动轮换信任锚。 在本文档中,我们描述了如何在不停机的情况下完成此操作。

独立于信任锚(trust anchor)颁发者证书(issuer certificate)密钥对(key pair)也可以过期(尽管可以 使用 cert-manager 设置自动轮换。 本文档还介绍了如何在不停机的情况下轮换颁发者证书密钥对

前提条件

这些说明使用 stepjq CLI 工具。

了解系统的当前状态

从运行以下命令开始:

linkerd check --proxy

如果您的配置有效并且您的凭据不会很快过期,您应该会看到类似于以下内容的输出:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

但是,如果您看到一条消息警告您您的信任锚(“trust root”)或颁发者证书即将到期,那么您必须轮换它们。

请注意,本文档仅适用于信任根颁发者证书当前有效的情况。 如果您的信任锚或颁发者证书已过期,请改为遵循 替换过期证书指南

例如,如果您的颁发者证书已过期,您将看到类似于以下内容的消息:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
× issuer cert is within its validity period
issuer certificate is not valid anymore. Expired on 2019-12-19T09:02:01Z
see https://linkerd.io/checks/#l5d-identity-issuer-cert-is-time-valid for hints

如果您的信任锚已过期,您将看到类似于以下内容的消息:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
× trust roots are within their validity period
Invalid roots:
* 79461543992952791393769540277800684467 identity.linkerd.cluster.local not valid anymore. Expired on 2019-12-19T09:11:30Z
see https://linkerd.io/checks/#l5d-identity-roots-are-time-valid  for hints

轮换信任锚

在不停机的情况下轮换信任锚是一个多步骤的过程:您必须生成一个新的信任锚, 将其与旧信任锚捆绑在一起,轮换颁发者证书密钥对, 最后从捆绑中删除旧信任锚。如果您只需要轮换颁发者证书和密钥对, 则可以直接跳到轮换身份颁发者证书 并忽略信任锚轮换步骤。

生成新的信任锚

首先,生成一个新的信任锚证书和私钥:

step certificate create root.linkerd.cluster.local ca-new.crt ca-new.key --profile root-ca --no-password --insecure

请注意,我们使用 --no-password --insecure 来避免使用 passphrase 对这些文件进行加密。 将私钥存储在安全的地方,以便将来可以使用它来 生成新的颁发者证书

将您的原始信任锚与新信任锚捆绑在一起

接下来,我们需要将 Linkerd 当前使用的信任锚与新锚捆绑在一起。 以下命令使用 kubectl 获取 Linkerd 配置, jq/yq 提取当前信任锚, 并且 step 将其与新生成的信任锚合并:

kubectl -n linkerd get cm linkerd-config -o=jsonpath='{.data.values}' \
  | yq e .identityTrustAnchorsPEM - > original-trust.crt

step certificate bundle ca-new.crt original-trust.crt bundle.crt
rm original-trust.crt

将新 bundle 部署到 Linkerd

此时你可以使用 linkerd upgrade 命令来指示 Linkerd 使用新的信任包:

linkerd upgrade --identity-trust-anchors-file=./bundle.crt | kubectl apply -f -

或者你也可以使用 helm upgrade 命令:

helm upgrade linkerd2 --set-file identityTrustAnchorsPEM=./bundle.crt

这将重新启动 Linkerd 控制平面中的代理,并将使用新的信任锚重新配置它们。

最后,您必须为集群中所有注入的工作负载重新启动代理。 例如,对 emojivoto 命名空间执行此操作将如下所示:

kubectl -n emojivoto rollout restart deploy

现在您可以运行 check 命令以确保一切正常:

linkerd check --proxy

您可能需要稍等片刻,直到所有 pod 都重新启动并配置了正确的信任锚。同时,您可能会观察到警告:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2019-12-19T09:51:19Z
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
‼ data plane proxies certificate match CA
    Some pods do not have the current trust bundle and must be restarted:
        * emojivoto/emoji-d8d7d9c6b-8qwfx
        * emojivoto/vote-bot-588499c9f6-zpwz6
        * emojivoto/voting-8599548fdc-6v64k
        * emojivoto/web-67c7599f6d-xx98n
        * linkerd/linkerd-sp-validator-75f9d96dc-rch4x
        * linkerd/linkerd-tap-68d8bbf64-mpzgb
        * linkerd/linkerd-web-849f74b7c6-qlhwc
    see https://linkerd.io/checks/#l5d-identity-data-plane-proxies-certs-match-ca for hints

当部署完成时,您的 check 命令应该停止警告您需要重新启动 pod。 但是,它可能仍会警告您,您的颁发者证书即将到期:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2019-12-19T09:51:19Z
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

轮换身份颁发者证书

要轮换颁发者证书和密钥对,首先生成一个新对:

step certificate create identity.linkerd.cluster.local issuer-new.crt issuer-new.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca-new.crt --ca-key ca-new.key

如果信任锚没有过期,并且如果最近轮换了所有代理, 则所有代理都已更新为包括一个工作信任锚(如上一节所述), 现在可以安全地再次使用 upgrade 命令轮换身份颁发者证书:

linkerd upgrade --identity-issuer-certificate-file=./issuer-new.crt --identity-issuer-key-file=./issuer-new.key | kubectl apply -f -

或:

exp=$(cat ca-new.crt | openssl x509 -noout -dates | grep "notAfter" | sed -e 's/notAfter=\(.*\)$/"\1"/' | TZ='GMT' xargs -I{} date -d {} +"%Y-%m-%dT%H:%M:%SZ")

helm upgrade linkerd2
  --set-file identity.issuer.tls.crtPEM=./issuer-new.crt
  --set-file identity.issuer.tls.keyPEM=./issuer-new.key
  --set identity.issuer.crtExpiry=$exp

此时,Linkerdidentity 控制平面服务应检测 到 secret 的更改并自动更新其颁发者证书。

为确保发生这种情况,您可以检查特定的 Kubernetes 事件:

kubectl get events --field-selector reason=IssuerUpdated -n linkerd

LAST SEEN   TYPE     REASON          OBJECT                        MESSAGE
9s          Normal   IssuerUpdated   deployment/linkerd-identity   Updated identity issuer

为集群中所有注入的工作负载重新启动代理,以确保它们的代理获取新颁发者颁发的证书:

kubectl -n emojivoto rollout restart deploy

运行 check 命令以确保一切按预期进行:

linkerd check --proxy

您应该看到没有任何证书过期警告的输出(除非仍然需要删除过期的信任锚):

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

删除旧的信任锚

我们现在可以从之前创建的信任 bundle 中删除旧的信任锚。 upgrade 命令可以为 Linkerd 组件执行此操作:

linkerd upgrade  --identity-trust-anchors-file=./ca-new.crt  | kubectl apply -f -

helm upgrade linkerd2 --set-file --set-file identityTrustAnchorsPEM=./ca-new.crt

请注意,./ca-new.crt 文件与您在此过程开始时创建的信任锚相同。 此外,您可以使用 rollout restart 命令更新其他注入资源的配置:

kubectl -n emojivoto rollout restart deploy
linkerd check --proxy

最后,check 命令的输出不应产生任何警告或错误:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

恭喜,您已轮换信任锚! 🎉