kubeadm使用外部etcd集群tls部署kubernetes

环境：ubuntu 16.04.2

             cpu 4  内存 8G

             内核4.4.0-119

ip地址：192.168.0.62

                 192.168.0.63

                 192.168.0.64

 etcd版本： 3.2.12

 kubernetes版本：1.11.5

 一、部署etcd集群(需要sudo或者root权限)

 1生成证书及etcd的二进制文件包，工具下载地址

 wget -O /bin/cfssl  https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 

wget -O /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 

chmod +x /bin/cfssl*

wget https://github.com/etcd-io/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz

tar xf etcd-v3.2.18-linux-amd64.tar.gz

cp etcd-v3.2.18-linux-amd64/etcd* /usr/bin/

2.ca-config.json配置文件，修改过期时间为10年（红色部分）

内容如下：

{

    "signing": {

        "default": {

            "expiry": "87600h"

        },

        "profiles": {

            "etcd": {

                "expiry": "87600h",

                "usages": [

                    "signing",

                    "key encipherment",

                    "client auth",

                    "server auth"

                ]

            }

        }

    }

}

3.ca-csr.json配置文件如下：

{

  "CN": "etcd",

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "shanghai",

      "L": "shanghai",

      "O": "etcd",

      "OU": "System"

    }

  ]

}

4.etcd集群的etcd-csr.json

{

  "CN": "etcd",

  "hosts": [

    "127.0.0.1",

    "192.168.0.62",

    "192.168.0.63",

    "192.168.0.64"

  ],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "shanghai",

      "L": "shanghai",

      "O": "etcd",

      "OU": "System"

    }

  ]

}

4.生成证书并自签名

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd

复制pem文件到你指定的目录，3台主机都要复制的

不建议使用目录/etc/kubernetes/pki/etcd

mkdir -p /etc/etcdCA

cp *.pem /etc/etcdCA

5.etcd的配置文件如下，红色部分自行更改。复制配置文件为/etc/default/etcd

ETCD_NAME=test-node62

ETCD_DATA_DIR="/var/lib/etcd/"

ETCD_LISTEN_PEER_URLS="https://192.168.0.62:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.0.62:2379,https://127.0.0.1:4001"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.62:2380"

ETCD_INITIAL_CLUSTER="test-node62=https://192.168.0.62:2380,test-node63=https://192.168.0.63:2380,test-node64=https://192.168.0.64:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-sdn"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.62:2379"

CLIENT_CERT_AUTH="true"

ETCD_CA_FILE="/etc/etcdCA/ca.pem"

ETCD_CERT_FILE="/etc/etcdCA/etcd.pem"

ETCD_KEY_FILE="/etc/etcdCA/etcd-key.pem"

PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_CA_FILE="/etc/etcdCA/ca.pem"

ETCD_PEER_CERT_FILE="/etc/etcdCA/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcdCA/etcd-key.pem"

6.创建用户和服务并授权

useradd etcd

chmod 755 /etc/etcdCA/*

echo '[Unit]

Description=etcd - highly-available key value store

Documentation=https://github.com/coreos/etcd

Documentation=man:etcd

After=network.target

Wants=network-online.target

[Service]

Environment=DAEMON_ARGS=

Environment=ETCD_NAME=%H

Environment=ETCD_DATA_DIR=/var/lib/etcd/default

EnvironmentFile=-/etc/default/%p

Type=notify

User=etcd

PermissionsStartOnly=true

#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"

ExecStart=/usr/bin/etcd $DAEMON_ARGS

Restart=on-abnormal

#RestartSec=10s

#LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

Alias=etcd3.service'   >/lib/systemd/system/etcd.service

7.启动服务

systemctl start etcd

8.检查集群状态

export ETCDCTL_API=3
etcdctl  \
  --cacert=/etc/etcdCA/ca.pem \
  --cert=/etc/etcdCA/etcd.pem \
  --key=/etc/etcdCA/etcd-key.pem \
  --endpoints=192.168.0.62:2379,192.168.0.63:2379,192.168.0.64:2379 \
  endpoint health

看到下图就ok了

二、部署kubernetes

1. 安装docker-ce （18.06.3）

sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce=18.06.3~ce~3-0~ubuntu

2.安装kubernetes包

apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get install -y kubelet=1.11.5-00 kubeadm=1.11.5-00 kubectl=1.11.5-00

3.使用配置文件进行安装，配置文件（kubeadm-config.yaml）如下

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
networking:
  podSubnet: 172.16.0.0/16
  serviceSubnet: 10.96.0.0/12
etcd:
  endpoints:
  - https://192.168.0.62:2379
  - https://192.168.0.63:2379
  - https://192.168.0.64:2379
  caFile: /etc/etcdCA/ca.pem
  certFile: /etc/etcdCA/etcd.pem
  keyFile: /etc/etcdCA/etcd-key.pem
kubernetesVersion: v1.11.5
kubeProxy:
  config:
    mode: "ipvs"

4.加载需要的kubernetes镜像

A="kube-proxy-amd64:v1.11.5
kube-apiserver-amd64:v1.11.5
kube-controller-manager-amd64:v1.11.5
kube-scheduler-amd64::v1.11.5
pause:3.1"
for i in $A;do
	docker pull mirrorgooglecontainers/$i
	docker tag mirrorgooglecontainers/$i k8s.gcr.io/$i
done
docker pull coredns/coredns:1.1.3
docker tag coredns/coredns:1.1.3 k8s.gcr.io/coredns:1.1.3

5.安装master,出现下图就master的安装好了

kubeadm init --config /path/kubeadm-config.yaml

6.授权客户端访问

  mkdir -p $HOME/.kube

  sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config

  sudo chown $(id -u):$(id -g) $HOME/.kube/config

7.安装客户端（请先执行1.安装docker-ce 2.kubernetes包和4.加载需要的kubernetes镜像）

执行master生成后的kubeadm jion ，需要root或者sudo权限

如上图是：

kubeadm join 192.168.0.62:6443 --token 4msj6v.plj3rcsq89c4y4mn --discovery-token-ca-cert-hash sha256:7fb655510bc0af2dda7e401a45932709c473b0f33acef0794924b54715512bbc

三、安装calico插件

wget https://github.com/projectcalico/calico/releases/download/v2.6.12/release-v2.6.12.tgz
tar xf release-v2.6.12.tgz
cd release-v2.6.12/k8s-manifests/hosted
sed -i 's?http://127.0.0.1:2379?https://192.168.0.62:2379,https://192.168.0.63:2379,https://192.168.0.64:2379?g' calico.yaml
cat /etc/etcdCA/etcd-key.pem|base64 -w 0 > ETCD-KEY
cat /etc/etcdCA/ca.pem|base64 -w 0 > ETCD-CA
cat /etc/etcdCA/etcd.pem|base64 -w 0 > ETCD-CERT
sed -i "s?# etcd-key: null?etcd-key: $(cat ETCD-KEY)?g" calico.yaml
sed -i "s?# etcd-ca: null?etcd-ca: $(cat ETCD-CA)?g" calico.yaml
sed -i "s?# etcd-cert: null?etcd-cert: $(cat ETCD-CERT)?g" calico.yaml
sed -i 's?etcd_ca: ""?etcd_ca: "/calico-secrets/etcd-ca"?g' calico.yaml
sed -i 's?etcd_cert: ""?etcd_cert: "/calico-secrets/etcd-cert"?g' calico.yaml
sed -i 's?etcd_key: ""?etcd_key: "/calico-secrets/etcd-key"?g' calico.yaml
kubectl apply -f calico.yaml
kubectl apply -f rbac-kdd.yaml

四、查看状态

至此k8s的基础部分完成

补充calico 3.10部分

wget https://github.com/projectcalico/calico/releases/download/v3.10.2/release-v3.10.2.tgz
tar xf release-v3.10.2.tgz
cd release-v3.10.2/k8s-manifests
sed -i 's?http://<ETCD_IP>:<ETCD_PORT>?https://192.168.0.62:2379,https://192.168.0.63:2379,https://192.168.0.64:2379?g' calico-etcd.yaml
cat /etc/etcdCA/etcd-key.pem|base64 -w 0 > ETCD-KEY
cat /etc/etcdCA/ca.pem|base64 -w 0 > ETCD-CA
cat /etc/etcdCA/etcd.pem|base64 -w 0 > ETCD-CERT
sed -i "s?# etcd-key: null?etcd-key: $(cat ETCD-KEY)?g" calico-etcd.yaml
sed -i "s?# etcd-ca: null?etcd-ca: $(cat ETCD-CA)?g" calico-etcd.yaml
sed -i "s?# etcd-cert: null?etcd-cert: $(cat ETCD-CERT)?g" calico-etcd.yaml
sed -i 's?etcd_ca: ""?etcd_ca: "/calico-secrets/etcd-ca"?g' calico-etcd.yaml
sed -i 's?etcd_cert: ""?etcd_cert: "/calico-secrets/etcd-cert"?g' calico-etcd.yaml
sed -i 's?etcd_key: ""?etcd_key: "/calico-secrets/etcd-key"?g' calico-etcd.yaml
sed -i 's?192.168.0.0/16?172.16.0.0/16?g' calico-etcd.yaml
kubectl apply -f calico-etcd.yaml

注意：加密的etcd集群和明文的etcd集群不能通用