如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

发布时间：2021-09-28 14:58:38 来源：亿速云阅读：138 作者：柒染栏目：系统运维

本篇文章为大家展示了如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群，内容简明扼要并且容易理解，绝对能使你眼前一亮，通过这篇文章的详细介绍希望你能有所收获。

服务器规划：

至少三台服务器组建高可用集群，配置2C4G以上

master，node节点均已安装docker，版本18以上

VIP 172.30.2.60

172.30.0.109 k8s-master1 nginx keepalived

172.30.0.89 k8s-master2 nginx keepalived

172.30.0.81 k8s-node1

二进制Kubernetes安装路径：/opt/kubernetes/{ssl,cfg,bin,logs}分别存放密钥，配置，可执行，日志文件

二进制etcd安装路径：/opt/etcd/{ssl,cfg,bin}分别存放密钥，配置，可执行文件

1、系统初始化

1.关闭防火墙：

# systemctl stop firewalld

# systemctl disable firewalld

2.关闭selinux：

# setenforce 0 # 临时

# sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久

3.关闭swap：

# swapoff -a # 临时

# vim /etc/fstab # 永久

4.同步系统时间：

# ntpdate time.windows.com

5.添加hosts：

# vim /etc/hosts

172.30.0.109 k8s-master1

172.30.0.81 k8s-master2

172.30.0.89 k8s-node1

6.修改主机名：

hostnamectl set-hostname k8s-master1

2、Etcd集群

1.安装cfssl工具

# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
# mv cfssl_linux-amd64 /usr/local/bin/cfssl
# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.生成etcd证书

①生成server.csr.json 修改请求文件中hosts字段包含所有etcd节点IP：

# vi server-csr.json

{

"CN": "etcd",

"hosts": [

"etcd01节点IP",

"etcd02节点IP",

"etcd03节点IP"

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing"

}

]

}

②生成自定义CA，初始化配置文件

# vim ca-config.json

{

"signing": {

"default": {

"expiry": "87600h"

"profiles": {

"www": {

"expiry": "87600h",

"usages": [

"signing",

"key encipherment",

"server auth",

"client auth"

]

}

# vim ca-csr.json

{

"CN": "etcd CA",

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "Beijing",

"ST": "Beijing"

}

]

}

etcd密钥初始化文件准备好后，即可生成密钥

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

生成的密钥放置在指定的etcd/ssl目录下

etcd二进制可执行文件etcd，etcdctl放置在etcd/bin下

etcd配置文件放置在etcd/cfg下，配置文件如下

# cat /opt/etcd/cfg/etcd

#[Member]

ETCD_NAME="etcd02"

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="https://172.30.0.81:2380"

ETCD_LISTEN_CLIENT_URLS="https://172.30.0.81:2379"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.30.0.81:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://172.30.0.81:2379"

ETCD_INITIAL_CLUSTER="etcd01=https://172.30.2.10:2380,etcd02=https://172.30.0.81:2380,etcd03=https://172.30.0.89:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

将etcd.service放置在/usr/lib/systemd/system/下

# cat /usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

EnvironmentFile=-/opt/etcd/cfg/etcd

ExecStart=/opt/etcd/bin/etcd \

--name=${ETCD_NAME} \

--data-dir=${ETCD_DATA_DIR} \

--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \

--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \

--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \

--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \

--initial-cluster=${ETCD_INITIAL_CLUSTER} \

--initial-cluster-token=${ETCD_INITIAL_CLUSTER} \

--initial-cluster-state=new \

--cert-file=/opt/etcd/ssl/server.pem \

--key-file=/opt/etcd/ssl/server-key.pem \

--peer-cert-file=/opt/etcd/ssl/server.pem \

--peer-key-file=/opt/etcd/ssl/server-key.pem \

--trusted-ca-file=/opt/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

分别拷贝到Etcd三个节点：

# scp –r /opt/etcd root@etcd节点IP:/opt

# scp /usr/lib/systemd/system/etcd.service root@etcd节点IP:/usr/lib/systemd/system

同步修改etcd节点配置文件的配置名称，使其与主机相对应，如

如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

启动etcd节点，etcd安装完成

# systemctl start etcd && systemctl enable etcd

检查etcd节点健康状态

# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem -- endpoints="https://172.30.2.10:2379,https://172.30.0.81:2379,https://172.30.0.89:2379" cluster-health

member 37f20611ff3d9209 is healthy: got healthy result from https://172.30.2.10:2379

member b10f0bac3883a232 is healthy: got healthy result from https://172.30.0.81:2379

member b46624837acedac9 is healthy: got healthy result from https://172.30.0.89:2379

cluster is healthy

3、部署K8S集群——master节点

①重新生成自定义CA，与etcd进行区分，生成apiserver，kube-proxy，kubectl-admin证书

# vim ca-config.json

{

"signing": {

"default": {

"expiry": "87600h"

"profiles": {

"kubernetes": {

"expiry": "87600h",

"usages": [

"signing",

"key encipherment",

"server auth",

"client auth"

]

}

# vim ca-csr.json

{

"CN": "kubernetes",

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "Beijing",

"ST": "Beijing",

"O": "k8s",

"OU": "System"

}

]

}

②生成kube-proxy密钥初始化文件

cat kube-proxy-csr.json

{

"CN": "system:kube-proxy",

"hosts": [],

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "k8s",

"OU": "System"

}

]

}

③生成apiserver密钥初始化文件，注意需要写入所有master节点IP，以及需要访问apiserver的IP地址，包括nginx，nginx vip，否则需要重新生成密钥

# vim server-csr.json

{

"CN": "kubernetes",

"hosts": [

"10.0.0.1",

"127.0.0.1",

"kubernetes",

"kubernetes.default",

"kubernetes.default.svc",

"kubernetes.default.svc.cluster",

"kubernetes.default.svc.cluster.local",

"172.30.2.60",

"172.30.0.109",

"172.30.0.81",

"172.30.2.10",

"172.30.0.89"

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "k8s",

"OU": "System"

}

]

}

④admin密钥，供kubectl远程客户端运行

# vim admin-csr.json

{

"CN": "admin",

"hosts": [],

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "system:masters",

"OU": "System"

}

]

}

⑤生成密钥

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

同步*.pem密钥至所有节点的/opt/kubernetes/ssl（也可以针对性同步密钥），kubelet密钥是后面部署kubelet自动生成的

如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

生成admin的kubeconfig文件，用于客户端访问k8s集群

# kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://172.30.0.109:6443 --kubeconfig=/root/.kube/kubectl.kubeconfig

# kubectl config set-credentials kube-admin --client-certificate=/opt/kubernetes/ssl/admin.pem --client-key=/opt/kubernetes/ssl/admin-key.pem --embed-certs=true --kubeconfig=/root/.kube/kubectl.kubeconfig

# kubectl config set-context kube-admin@kubernetes --cluster=kubernetes --user=kube-admin --kubeconfig=/root/.kube/kubectl.kubeconfig

# kubectl config use-context kube-admin@kubernetes --kubeconfig=/root/.kube/kubectl.kubeconfig

# mv /root/.kube/{kubectl.kubeconfig,config}

⑥下载master二进制包放置在/opt/kubernetes/bin下

如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

⑦配置配置文件，service启动文件

kube-apiserve 注意：部署另一台master时，需要更换IP地址

# vim /opt/kubernetes/cfg/kube-apiserver.conf

KUBE_APISERVER_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--etcd-servers=https://172.30.2.10:2379,https://172.30.0.81:2379,https://172.30.0.89:2379 \

--bind-address=172.30.0.109 \

--secure-port=6443 \

--advertise-address=172.30.0.109 \

--allow-privileged=true \

--service-cluster-ip-range=10.0.0.0/24 \

--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \

--authorization-mode=RBAC,Node \

--enable-bootstrap-token-auth=true \

--token-auth-file=/opt/kubernetes/cfg/token.csv \

--service-node-port-range=30000-32767 \

--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \

--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \

--tls-cert-file=/opt/kubernetes/ssl/server.pem \

--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \

--client-ca-file=/opt/kubernetes/ssl/ca.pem \

--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \

--etcd-cafile=/opt/etcd/ssl/ca.pem \

--etcd-certfile=/opt/etcd/ssl/server.pem \

--etcd-keyfile=/opt/etcd/ssl/server-key.pem \

--audit-log-maxage=30 \

--audit-log-maxbackup=3 \

--audit-log-maxsize=100 \

--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

# cat /usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf

ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

kube-controller-manager

# vim /opt/kubernetes/cfg/kube-controller-manager.conf

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--leader-elect=true \

--master=127.0.0.1:8080 \

--address=127.0.0.1 \

--allocate-node-cidrs=true \

--cluster-cidr=10.244.0.0/16 \

--service-cluster-ip-range=10.0.0.0/24 \

--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \

--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \

--root-ca-file=/opt/kubernetes/ssl/ca.pem \

--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \

--experimental-cluster-signing-duration=87600h0m0s"

# vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf

ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

kube-scheduler

# vim /opt/kubernetes/cfg/kube-scheduler.conf

KUBE_SCHEDULER_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--leader-elect \

--master=127.0.0.1:8080 \

--address=127.0.0.1"

# vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]

Description=Kubernetes Scheduler

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf

ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

生成token文件，供node节点与apiserver进行通信，注意token值，master需要与node保持一致

# cat /opt/kubernetes/cfg/token.csv

c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"

格式：token,用户,uid,用户组

token也可自行生成替换：

# head -c 16 /dev/urandom | od -An -t x | tr -d ' '

授权kubelet-bootstrap，使node节点kubelet能够正常访问到apiserver

# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

启动master节点，观察日志/opt/kubernetes/logs

# systemctl start kube-controller-manager

# systemctl start kube-scheduler

# systemctl enable kube-apiserver

# systemctl enable kube-controller-manager

# systemctl enable kube-scheduler

原样部署第二个master节点172.30.0.81，不同之处仅在于apiserver配置文件当中的IP地址

4、部署K8S集群——node节点

①node节点密钥准备

已提前同步至node节点的/opt/kubernetes/ssl下

②配置node节点kubelet，kube-proxy配置文件

kubelet 注意：不同节点的kubelet需要修改hostname配置

# vim kubelet.conf

KUBELET_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--hostname-override=k8s-node1 \

--network-plugin=cni \

--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \

--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \

--config=/opt/kubernetes/cfg/kubelet-config.yml \

--cert-dir=/opt/kubernetes/ssl \

--pod-infra-container-image=lizhenliang/pause-amd64:3.0"

bootstrap.kubeconfig 这个文件是kubelet与apiserver的通信认证文件，内部token需要与master节点的token文件保持一致

#vim bootstrap.kubeconfig

apiVersion: v1

clusters:

- cluster:

certificate-authority: /opt/kubernetes/ssl/ca.pem

server: https://172.30.0.109:6443

contexts:

- context:

cluster: kubernetes

user: kubelet-bootstrap

current-context: default

kind: Config

preferences: {}

users:

- name: kubelet-bootstrap

user:

token: c47ffb939f5ca36231d9e3121a252940

# vim kubelet-config.yml

kind: KubeletConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1

address: 0.0.0.0

port: 10250

readOnlyPort: 10255

cgroupDriver: cgroupfs

clusterDNS:

- 10.0.0.2

clusterDomain: cluster.local

failSwapOn: false

authentication:

anonymous:

enabled: false

webhook:

cacheTTL: 2m0s

enabled: true

x509:

clientCAFile: /opt/kubernetes/ssl/ca.pem

authorization:

mode: Webhook

webhook:

cacheAuthorizedTTL: 5m0s

cacheUnauthorizedTTL: 30s

evictionHard:

imagefs.available: 15%

memory.available: 100Mi

nodefs.available: 10%

nodefs.inodesFree: 5%

maxOpenFiles: 1000000

maxPods: 110

# vim /usr/lib/systemd/system/kubelet.service

[Unit]

Description=Kubernetes Kubelet

After=docker.service

Before=docker.service

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf

ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

kube-proxy

# vim kube-proxy.conf

KUBE_PROXY_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--config=/opt/kubernetes/cfg/kube-proxy-config.yml"

# vim kube-proxy-config.yml

kind: KubeProxyConfiguration

apiVersion: kubeproxy.config.k8s.io/v1alpha1

address: 0.0.0.0

metricsBindAddress: 0.0.0.0:10249

clientConnection:

kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig

hostnameOverride: k8s-node1

clusterCIDR: 10.0.0.0/24

mode: ipvs

ipvs:

scheduler: "rr"

iptables:

masqueradeAll: true

kube-proxy.kubeconfig kube-proxy通信认证文件

# vim kube-proxy.kubeconfig

apiVersion: v1

clusters:

- cluster:

certificate-authority: /opt/kubernetes/ssl/ca.pem

server: https://172.30.0.109:6443

contexts:

- context:

cluster: kubernetes

user: kube-proxy

current-context: default

kind: Config

preferences: {}

users:

- name: kube-proxy

user:

client-certificate: /opt/kubernetes/ssl/kube-proxy.pem

client-key: /opt/kubernetes/ssl/kube-proxy-key.pem

# vim /usr/lib/systemd/system/kube-proxy.service

[Unit]

Description=Kubernetes Proxy

After=network.target

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf

ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

/opt/kubernetes/cfg配置如下kubelet.kubeconfig是启动后自动生成的

如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

/opt/kubernetes/bin

如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

③启动node节点

# systemctl start kubelet

# systemctl start kube-proxy

# systemctl enable kubelet

# systemctl enable kube-proxy

④允许给node颁发证书

# kubectl get csr

# kubectl certificate approve node-csr-MYUxbmf_nmPQjmH3LkbZRL2uTO-_FCzDQUoUfTy7YjI

# kubectl get node

5、部署CNI网络

二进制包下载地址：https://github.com/containernetworking/plugins/releases

# mkdir /opt/cni/bin /etc/cni/net.d

# tar zxvf cni-plugins-linux-amd64-v0.8.2.tgz –C /opt/cni/bin

确保kubelet启用CNI：

# cat /opt/kubernetes/cfg/kubelet.conf

--network-plugin=cni

在Master执行：

# kubectl apply -f kube-flannel.yaml

# kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE

kube-flannel-ds-amd64-5xmhh 1/1 Running 6 171m

kube-flannel-ds-amd64-ps5fx 1/1 Running 0 150m

6、授权apiserver访问kubelet

为提供安全性，kubelet禁止匿名访问，必须授权才可以。

# vim apiserver-to-kubelet-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

annotations:

rbac.authorization.kubernetes.io/autoupdate: "true"

labels:

kubernetes.io/bootstrapping: rbac-defaults

rules:

- apiGroups:

- ""

resources:

- nodes/proxy

- nodes/stats

- nodes/log

- nodes/spec

- nodes/metrics

- pods/log

verbs:

- "*"

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

namespace: ""

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

- apiGroup: rbac.authorization.k8s.io

kind: User

# kubectl apply -f apiserver-to-kubelet-rbac.yaml

这样就能通过kubectl操作pod，查看日志

7、部署coredns

# kubectl apply –f coredns.yaml

8、K8S高可用配置

kube-controller-manager以及kube-scheduler高可用已经在部署集群的时候在配置中体现

选举进行负载均衡，所以我们只需要关注apiserver高可用即可

如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

kube-apiserver高可用

①首先配置好两台master节点

②在两台master节点上各自部署nginx，keepalived

keepalived监听nginx健康状态

nginx配置为四层监听两台master的6443端口，四层转发

nginx配置如下

# rpm -vih http://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.16.0-1.el7.ngx.x86_64.rpm

# vim /etc/nginx/nginx.conf

……

stream {

log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

access_log /var/log/nginx/k8s-access.log main;

upstream k8s-apiserver {

server 172.30.0.109:6443;

server 172.30.0.81:6443;

}

server {

listen 6443;

proxy_pass k8s-apiserver;

}

……

启动nginx

# systemctl start nginx

# systemctl enable nginx

keepliaved vip配置为172.30.2.60

③配置node节点访问master apiserver经由172.30.2.60进行四层转发

修改node节点配置，将单点的master的节点，更改为VIP 172.30.2.60即可

如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

批量修改：

# sed -i 's#172.30.0.109#172.30.2.60#g' *

上述内容就是如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群，你们学到知识或技能了吗？如果还想学到更多技能或者丰富自己的知识储备，欢迎关注亿速云行业资讯频道。

向AI问一下细节

如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群

猜你喜欢

最新资讯

相关推荐

相关标签