1、k8s 集群架构描述

  • kubeadm v1.10.2创建k8s集群。
  • master节点高可用,三节点(10.18.60.3、10.18.60.4、10.18.60.5)。
  • LVS实现master三节点代理。

 

2、K8S集群证书过期,日志报错如下

1
Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid

 

3、故障排定,查看证书

1
2
3
# openssl x509 -noout -text -in apiserver-kubelet-client.crt | grep Not
            Not Before: May 22 01:58:06 2018 GMT
            Not After : May 22 01:58:07 2019 GMT    # 2019-5-22日过期

 

4、kubeadm 命令介绍(v1.10.2) 

注:本次只需用到以下两个参数命令,其它参数不做介绍

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# kubeadm alpha phase certs -h        # 创建证书
Usage:
  kubeadm alpha phase certs [command]
Available Commands:
  all                      Generates all PKI assets necessary to establish the control plane
  apiserver                Generates an API server serving certificate and key
  apiserver-etcd-client    Generates a client certificate for the API server to connect to etcd securely
  apiserver-kubelet-client Generates a client certificate for the API server to connect to the kubelets securely
  ca                       Generates a self-signed kubernetes CA to provision identities for components of the cluster
  etcd-ca                  Generates a self-signed CA to provision identities for etcd
  etcd-healthcheck-client  Generates a client certificate for liveness probes to healthcheck etcd
  etcd-peer                Generates an etcd peer certificate and key
  etcd-server              Generates an etcd serving certificate and key
  front-proxy-ca           Generates a front proxy CA certificate and key for a Kubernetes cluster
  front-proxy-client       Generates a front proxy CA client certificate and key for a Kubernetes cluster
  sa                       Generates a private key for signing service account tokens along with its public key# kubeadm alpha phase kubeconfig -h    # 生成配置文件(例如:admin.conf|controller-manager.conf|kubelet.conf|scheduler.conf)
Usage:
  kubeadm alpha phase kubeconfig [command]
Available Commands:
  admin              Generates a kubeconfig file for the admin to use and for kubeadm itself
  all                Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
  controller-manager Generates a kubeconfig file for the controller manager to use
  kubelet            Generates a kubeconfig file for the kubelet to use. Please note that this should be used *only* for bootstrapping purposes.
  scheduler          Generates a kubeconfig file for the scheduler to use
  user               Outputs a kubeconfig file for an additional user
# kubeadm alpha phase certs apiserver -h
      --apiserver-advertise-address string   填写本机apiserver ip。
      --apiserver-cert-extra-sans strings    多master节点,在创建apiserver证书时,需要指定每个节点的IP,代理IP、域名。
      --cert-dir string                      The path where to save the certificates (default "/etc/kubernetes/pki")
      --config string                        Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
      -h, --help                                 help for apiserver
      --service-cidr string                  Alternative range of IP address for service VIPs, from which derives the internal API server VIP that will be added to the API Server serving cert (default "10.96.0.0/12")
      --service-dns-domain string            Alternative domain for services, to use for the API server serving cert (default "cluster.local")  

 

5、备份节点配置文件与证书

# cp -rfp /etc/kubernetes /etc/kubernetes.2019.5.23

6、创建证书

注:

    1、因为之前三个master节点的配置文件中全部填写的是LVS VIP(没做域名解析),为了今后切换方便给VIP配置了一个域名,而apiserver证书中没有配置该域名的认证,所以利用openssl对apiserver证书做了重签替换,并设置apiserver证书有效期10年,所以这次证书过期不涉及apiserver,只需要对apiserver-kubelet-client与front-proxy-client证书重新创建,而下边给出了对apiserver证书重签的命令。

2、创建证书时需要配置VPN,kubeadm需要连接国外

1
2
3
# kubeadm alpha phase certs apiserver --apiserver-advertise-address 10.18.60.3 --apiserver-cert-extra-sans 10.18.60.4 --apiserver-cert-extra-sans 10.18.60.5 --apiserver-cert-extra-sans 10.16.60.6 --apiserver-cert-extra-sans k8s.m.api   # 创建apiserver证书
# kubeadm alpha phase certs apiserver-kubelet-client                  # 创建apiserver-kubelet-client证书
# kubeadm alpha phase certs front-proxy-client                    # 创建front-proxy-client证书

 

7、创建配置文件(admin.conf|controller-manager.conf|kubelet.conf|scheduler.conf)

1
2
# 创建完会看到/etc/kubernetes下面出现了配置文件
# kubeadm alpha phase kubeconfig all --apiserver-advertise-address 10.18.60.3 

 

8、准备替换

因为我三节点的代理IP配置了域名,所以需要做下替换。

1
2
3
4
5
6
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' admin.conf
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' controller-manager.conf
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' scheduler.conf
# sed -i 's/10\.5\.38\.39/k8s.m.api/g' kubelet.conf
# grep 'host:' /etc/kubernetes/manifests/kube-apiserver.yaml
        host: k8s.m.api

 

1
2
# 直接覆盖(注意文件权限)
# cp -rfp /etc/kubernetes/admin.conf ~/.kube/config

 

1
2
# kubelet 客户端签发的不需要备份
# rm -rf /var/lib/kubelet/pki/*

 

9、重启服务

1
2
3
4
5
# 重启本机所有docker容器
# docker restart $(docker ps -q)
# 重启kubelet
# systemctl restart kubelet.service

 

10、验证

1
2
# 可以看到已经恢复
# kubectl get node

 

11、恢复其它master节点

1
2
# 注意拷贝kubelet.conf文件到其它服务器(其它服务器自己生成)<br># scp admin.conf  controller-manager.conf  scheduler.conf 10.18.60.4:/etc/kubernetes
# scp admin.conf  controller-manager.conf  scheduler.conf 10.18.60.5:/etc/kubernetes<br># scp apiserver-kubelet-client.crt  apiserver-kubelet-client.key front-proxy-client.crt  front-proxy-client.key 10.18.60.4:/etc/kubernetes/pki<br># scp apiserver-kubelet-client.crt  apiserver-kubelet-client.key front-proxy-client.crt  front-proxy-client.key 10.18.60.5:/etc/kubernetes/pki

 

1
2
3
4
# 其它两个节点只生成各自的kubelet配置文件
# kubeadm alpha phase kubeconfig kubelet
# grep 'server:' kubelet.conf
    server: https://k8s.m.api:6443

 

1
2
# 直接覆盖(注意文件权限)
# cp -rfp /etc/kubernetes/admin.conf ~/.kube/config

 

1
2
# kubelet 客户端签发的不需要备份
# rm -rf /var/lib/kubelet/pki/*

 

1
2
3
4
5
# 重启本机所有docker容器
# docker restart $(docker ps -q)
# 重启kubelet
# systemctl restart kubelet.service

 

1
2
# 可以看到已经恢复
# kubectl get node

发表评论

您的电子邮箱地址不会被公开。 必填项已用*标注