二进制部署一套高可用K8s集群-v1.24+(一)
一、系统初始化
- 说明:本文档的角色规划和系统初始化流程跟下面链接中的文章规划一致,本文不在赘诉!
提示:
- 本文档使用的K8s版本为
1.24+
2.本文档使用的容器运行时为Containerd
3.本文档使用的网络插件为Calico
4.本文档使用的系统为CentOS 7.6
,内核版本5.4+
5.执行下面的操作之前,请确保K8s-master1节点机器与其它集群节点已经实现了主机名免密和IP免密登入
二、创建CA根证书和秘钥
1、安装cfssl工具集
[root@k8s-master1 ~]# cd /opt/k8s
[root@k8s-master1 k8s]# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
[root@k8s-master1 k8s]# mv cfssl_1.6.1_linux_amd64 /opt/k8s/bin/cfssl
[root@k8s-master1 k8s]# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
[root@k8s-master1 k8s]# mv cfssljson_1.6.1_linux_amd64 /opt/k8s/bin/cfssljson
[root@k8s-master1 k8s]# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64
[root@k8s-master1 k8s]# mv cfssl-certinfo_1.6.1_linux_amd64 /opt/k8s/bin/cfssl-certinfo
[root@k8s-master1 k8s]# chmod +x /opt/k8s/bin/*
[root@k8s-master1 k8s]# export PATH=/opt/k8s/bin:$PATH
[root@k8s-master1 k8s]# ls /opt/k8s/bin/
2、创建根证书(CA)
2.1:创建配置文件
[root@k8s-master1 ~]# cd /opt/k8s/work
[root@k8s-master1 work]# mkdir -p ca && cd ca
[root@k8s-master1 ca]# cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "876000h"
}
}
}
}
EOF
signing
:表示该证书可用于签名其它证书(生成的 ca.pem 证书中 CA=TRUE);
server auth
:表示 client 可以用该该证书对 server 提供的证书进行验证;
client auth
:表示 server 可以用该该证书对 client 提供的证书进行验证;
"expiry": "876000h"
:证书有效期设置为 100 年;
2.2:创建证书签名请求文件
[root@k8s-master1 ca]# cat > ca-csr.json <<EOF
{
"CN": "kubernetes-ca",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "k8s",
"OU": "dqz"
}
],
"ca": {
"expiry": "876000h"
}
}
EOF
2.3:生成CA证书和私钥
[root@k8s-master1 ca]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@k8s-master1 ca]# ls ca*
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
3、分发证书文件
将生成的 CA 证书、秘钥文件、配置文件拷贝到所有节点(master和worker节点)的
/etc/kubernetes/cert
目录下
[root@k8s-master1 ca]# for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /etc/kubernetes/cert"
scp ca*.pem ca-config.json root@${node_ip}:/etc/kubernetes/cert
done
[root@k8s-master1 ca]# for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "ls -lt /etc/kubernetes/cert"
done
三、部署ETCD集群
- etcd 是基于 Raft 的分布式 KV 存储系统,由 CoreOS 开发,常用于服务发现、共享配置以及并发控制(如 leader 选举、分布式锁等)。
- kubernetes 使用 etcd 集群持久化存储所有 API 对象、运行数据。
- etcd 集群节点名称和 IP 如下:
k8s-master1:192.168.66.62
k8s-master2:192.168.66.63
k8s-master3:192.168.66.64
1、下载和分发 etcd 二进制文件
ETCD仓库地址: https://github.com/etcd-io/etcd/releases
如果网络原因,请在本地下载好安装包并上传至服务器
1.1:解压安装
[root@k8s-master1 ~]# cd /opt/k8s/work/
[root@k8s-master1 work]# mkdir etcd && cd etcd
#下面的下载链接为加速地址
[root@k8s-master1 etcd]# wget https://github.91chi.fun/https://github.com//etcd-io/etcd/releases/download/v3.6.0-alpha.0/etcd-v3.6.0-alpha.0-linux-amd64.tar.gz
#解压包至当前目录下
[root@k8s-master1 etcd]# tar -xf etcd-v3.6.0-alpha.0-linux-amd64.tar.gz
1.2:分发各ETCD节点
[root@k8s-master1 etcd]# for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
scp etcd-v3.6.0-alpha.0-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
done
2、创建 etcd 证书和私钥
2.1:创建证书签名请求
注意:这里的IP地址一定要根据自己的实际ETCD集群IP填写;不然有可能会出现
error "remote error: tls: bad certificate", ServerName ""
的错误
[root@k8s-master1 ~]# cd /opt/k8s/work/etcd
[root@k8s-master1 etcd]# mkdir -p cert && cd cert/
[root@k8s-master1 cert]# cat > etcd-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.66.62",
"192.168.66.63",
"192.168.66.64"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "k8s",
"OU": "dqz"
}
]
}
EOF
2.2:生成证书和私钥
[root@k8s-master1 cert]# cfssl gencert -ca=/opt/k8s/work/ca/ca.pem \\
-ca-key=/opt/k8s/work/ca/ca-key.pem \\
-config=/opt/k8s/work/ca/ca-config.json \\
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
2.3:分发证书和私钥至各etcd节点
[root@k8s-master1 cert]# for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /etc/etcd/cert"
scp etcd*.pem root@${node_ip}:/etc/etcd/cert/
done
3、创建 etcd 的 systemd unit 模板文件
[root@k8s-master1 ~]# mkdir /opt/k8s/work/service-template
[root@k8s-master1 ~]# cd /opt/k8s/work/service-template
[root@k8s-master1 service-template]# mkdir -p etcd && cd etcd
[root@k8s-master1 etcd]# cat > etcd.service.template <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \\\\
--data-dir=${ETCD_DATA_DIR} \\\\
--wal-dir=${ETCD_WAL_DIR} \\\\
--name=##ETCD_NAME## \\\\
--cert-file=/etc/etcd/cert/etcd.pem \\\\
--key-file=/etc/etcd/cert/etcd-key.pem \\\\
--trusted-ca-file=/etc/kubernetes/cert/ca.pem \\\\
--peer-cert-file=/etc/etcd/cert/etcd.pem \\\\
--peer-key-file=/etc/etcd/cert/etcd-key.pem \\\\
--peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\\\
--peer-client-cert-auth \\\\
--client-cert-auth \\\\
--listen-peer-urls=https://##ETCD_IP##:2380 \\\\
--initial-advertise-peer-urls=https://##ETCD_IP##:2380 \\\\
--listen-client-urls=https://##ETCD_IP##:2379,http://127.0.0.1:2379 \\\\
--advertise-client-urls=https://##ETCD_IP##:2379 \\\\
--initial-cluster-token=etcd-cluster-0 \\\\
--initial-cluster=${ETCD_NODES} \\\\
--initial-cluster-state=new \\\\
--auto-compaction-mode=periodic \\\\
--auto-compaction-retention=1 \\\\
--max-request-bytes=33554432 \\\\
--quota-backend-bytes=6442450944 \\\\
--heartbeat-interval=250 \\\\
--election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
4、为各ETCD节点创建和分发 etcd systemd unit 文件
4.1:替换模板文件中的变量
[root@k8s-master1 etcd]# for (( i=0; i < 3; i++ ))
do
sed -e "s/##ETCD_NAME##/${ETCD_NAMES[i]}/" -e "s/##ETCD_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service
done
[root@k8s-master1 etcd]# ls *.service
etcd-192.168.66.62.service etcd-192.168.66.63.service etcd-192.168.66.64.service
4.2:分发生成的 systemd unit 文件
[root@k8s-master1 etcd]# for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
done
5、启动ETCD服务
- 必须创建 etcd 数据目录和工作目录;
- 注意:3.4.10+版本,需要将数据目录的权限设置为0700才可以正常启动
etcd 进程首次启动时会等待其它节点的 etcd 加入集群,命令 systemctl start etcd 会卡住一段时间,为正常现象。
注意:有可能ETCD节点1启动失败,而另外2个节点启动成功,这是正常情况,请重启ETCD节点1即可
[root@k8s-master1 etcd]# for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR} && chmod 0700 /data/k8s/etcd/data"
ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd"
done
#手动在master1节点运行启动ETCD服务
[root@k8s-master1 etcd]# systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd
6、检查启动结果
[root@k8s-master1 etcd]# for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "systemctl status etcd|grep Active"
done
7、验证服务状态
[root@k8s-master1 etcd]# for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
/opt/k8s/bin/etcdctl \\
--endpoints=https://${node_ip}:2379 \\
--cacert=/etc/kubernetes/cert/ca.pem \\
--cert=/etc/etcd/cert/etcd.pem \\
--key=/etc/etcd/cert/etcd-key.pem endpoint health
done
![](https://jkboy.com/wp-content/uploads/2021/10/Pasted-11.png)