国内环境kubernetes安装配置

环境

操作系统: CentOS Linux release 7.6.1810

内网ip: 192.168.20.15

kubernetes: v1.15.1

docker: 19.03.1

安装docker

1
2
3
4
curl -fsSL get.docker.com -o get-docker.sh
sh get-docker.sh --mirror Aliyun
systemctl enable docker
systemctl start docker

添加docker国内镜像

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cat <<EOF > /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"debug" : true,
"insecure-registries" : [
"nas.pocketdigi.com:8083"
],
"experimental" : false,
"registry-mirrors" : [
"https://registry.docker-cn.com",
"https://reg-mirror.qiniu.com"
]
}
EOF
systemctl restart docker
docker login nas.pocketdigi.com:8083

nas.pocketdigi.com:8083 是私有仓库地址,不支持https,所以需要放到insecure-registries,没有私有仓库就不需要

添加阿里云源,安装kubelet,kubeadm,kubectl

1
2
3
4
5
6
7
8
9
10
11
12
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

配置iptables

1
2
3
4
5
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

关闭swap

注释掉/etc/fstab文件里swap分区,重启

创建集群

1
kubeadm init --apiserver-advertise-address=192.168.20.15 --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=nas.pocketdigi.com

因为服务器没有外网ip,是通过nginx转发的,必须通过-apiserver-cert-extra-sans 配置最终访问的外网ip或域名

如果是国内服务器,因为k8s仓库被墙,会报以下错误:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'


error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.15.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.15.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.15.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.15.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.10: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1

好办,我们先找台海外服务器,把相应的镜像拉下来,推到我们自己的私有仓库里,再pull,然后改tag。没有私有仓库也不要紧,我已经把1.15.1推到hub.docker.com了。

找一台能连接k8s.gcr.io的服务器:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
docker pull k8s.gcr.io/kube-apiserver:v1.15.1
docker pull k8s.gcr.io/kube-controller-manager:v1.15.1
docker pull k8s.gcr.io/kube-scheduler:v1.15.1
docker pull k8s.gcr.io/kube-proxy:v1.15.1
docker pull k8s.gcr.io/pause:3.1
docker pull k8s.gcr.io/etcd:3.3.10
docker pull k8s.gcr.io/coredns:1.3.1

docker tag k8s.gcr.io/kube-apiserver:v1.15.1 nas.pocketdigi.com:8083/k8s.gcr.io/kube-apiserver:v1.15.1
docker tag k8s.gcr.io/kube-controller-manager:v1.15.1 nas.pocketdigi.com:8083/k8s.gcr.io/kube-controller-manager:v1.15.1
docker tag k8s.gcr.io/kube-scheduler:v1.15.1 nas.pocketdigi.com:8083/k8s.gcr.io/kube-scheduler:v1.15.1
docker tag k8s.gcr.io/kube-proxy:v1.15.1 nas.pocketdigi.com:8083/k8s.gcr.io/kube-proxy:v1.15.1
docker tag k8s.gcr.io/pause:3.1 nas.pocketdigi.com:8083/k8s.gcr.io/pause:3.1
docker tag k8s.gcr.io/etcd:3.3.10 nas.pocketdigi.com:8083/k8s.gcr.io/etcd:3.3.10
docker tag k8s.gcr.io/coredns:1.3.1 nas.pocketdigi.com:8083/k8s.gcr.io/coredns:1.3.1

docker push nas.pocketdigi.com:8083/k8s.gcr.io/kube-apiserver:v1.15.1
docker push nas.pocketdigi.com:8083/k8s.gcr.io/kube-controller-manager:v1.15.1
docker push nas.pocketdigi.com:8083/k8s.gcr.io/kube-scheduler:v1.15.1
docker push nas.pocketdigi.com:8083/k8s.gcr.io/kube-proxy:v1.15.1
docker push nas.pocketdigi.com:8083/k8s.gcr.io/pause:3.1
docker push nas.pocketdigi.com:8083/k8s.gcr.io/etcd:3.3.10
docker push nas.pocketdigi.com:8083/k8s.gcr.io/coredns:1.3.1

回到安装kubernetes的机器:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
docker pull nas.pocketdigi.com:8083/k8s.gcr.io/kube-apiserver:v1.15.1
docker pull nas.pocketdigi.com:8083/k8s.gcr.io/kube-controller-manager:v1.15.1
docker pull nas.pocketdigi.com:8083/k8s.gcr.io/kube-scheduler:v1.15.1
docker pull nas.pocketdigi.com:8083/k8s.gcr.io/kube-proxy:v1.15.1
docker pull nas.pocketdigi.com:8083/k8s.gcr.io/pause:3.1
docker pull nas.pocketdigi.com:8083/k8s.gcr.io/etcd:3.3.10
docker pull nas.pocketdigi.com:8083/k8s.gcr.io/coredns:1.3.1

docker tag nas.pocketdigi.com:8083/k8s.gcr.io/kube-apiserver:v1.15.1 k8s.gcr.io/kube-apiserver:v1.15.1
docker tag nas.pocketdigi.com:8083/k8s.gcr.io/kube-controller-manager:v1.15.1 k8s.gcr.io/kube-controller-manager:v1.15.1
docker tag nas.pocketdigi.com:8083/k8s.gcr.io/kube-scheduler:v1.15.1 k8s.gcr.io/kube-scheduler:v1.15.1
docker tag nas.pocketdigi.com:8083/k8s.gcr.io/kube-proxy:v1.15.1 k8s.gcr.io/kube-proxy:v1.15.1
docker tag nas.pocketdigi.com:8083/k8s.gcr.io/pause:3.1 k8s.gcr.io/pause:3.1
docker tag nas.pocketdigi.com:8083/k8s.gcr.io/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag nas.pocketdigi.com:8083/k8s.gcr.io/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

没有条件的小伙伴,请直接使用下面的命令下载我转到hub.docker.com的镜像:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
docker pull pocketdigi/kube-apiserver:v1.15.1
docker pull pocketdigi/kube-controller-manager:v1.15.1
docker pull pocketdigi/kube-scheduler:v1.15.1
docker pull pocketdigi/kube-proxy:v1.15.1
docker pull pocketdigi/pause:3.1
docker pull pocketdigi/etcd:3.3.10
docker pull pocketdigi/coredns:1.3.1

docker tag pocketdigi/kube-apiserver:v1.15.1 k8s.gcr.io/kube-apiserver:v1.15.1
docker tag pocketdigi/kube-controller-manager:v1.15.1 k8s.gcr.io/kube-controller-manager:v1.15.1
docker tag pocketdigi/kube-scheduler:v1.15.1 k8s.gcr.io/kube-scheduler:v1.15.1
docker tag pocketdigi/kube-proxy:v1.15.1 k8s.gcr.io/kube-proxy:v1.15.1
docker tag pocketdigi/pause:3.1 k8s.gcr.io/pause:3.1
docker tag pocketdigi/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag pocketdigi/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

重新执行init:

1
kubeadm init --apiserver-advertise-address=192.168.20.15 --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=nas.pocketdigi.com

–pod-network-cidr是因为我们后面用的是flannel网络插件,默认就是这个网段

成功后得到以下信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.20.15:6443 --token m5sli6.w3z0zprk0883acuo \
--discovery-token-ca-cert-hash sha256:114fd2e850b62e7cb9924be9fb980d75e10e3e4c8e2505eddbfdd2e0f081b964

注意保存token,token会在24小时后过期,到时如果需要往集群里加节点,需要重新创建token,discovery-token-ca-cert-hash值是不会变的

1
kubeadm token create

复制kubectl所需的配置文件:

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装网络插件flannel,先从镜像节点下载镜像

1
2
3
docker pull pocketdigi/flannel:v0.11.0-amd64
docker tag pocketdigi/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

允许在master节点安装pod(默认不允许)

1
kubectl taint nodes --all node-role.kubernetes.io/master-

后期如果增加了节点,要禁止在master节点部署pod,通过以下命令恢复:

1
kubectl taint nodes kubernetes-master  node-role.kubernetes.io/master=:NoSchedule

kubernetes-master是master节点名

如果使用外部安装的rancher管理集群,会无法注册,关闭防火墙解决:

1
systemctl disable firewalld

Nginx Ingress Controller

如果要对外暴露http服务,建议安装Ingress. 官方文档

官方文档里的yaml不会暴露80,443端口,需要下载后修改。

1
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

在nginx-ingress-controller Deployment的spec.template.spec节点下增加hostNetwork: true,即使用主机网络。
如果有多台机器,不只一个master节点,需要把Deployment改成DaemonSet,以便在每个节点都部署nginx。最后修改后如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses/status
verbs:
- update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
# replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
hostNetwork: true
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.0
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10

---

然后在master和worker节点pull镜像

1
2
docker pull pocketdigi/nginx-ingress-controller:0.25.0
docker tag pocketdigi/nginx-ingress-controller:0.25.0 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.0

在master节点执行:

1
kubectl apply -f mandatory.yaml

添加ingress-nginx Service,这一步不同主机提供商操作不同,参考官方文档,一般虚拟机就用下面的就可以了。

1
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml

导入SSL证书

1
kubectl create secret tls nas-pocketdigi-com --key private.pem  --cert fullchain.pem --namespace prod

nas-pocketdigi-com是证书名,我这个是泛域名证书,prod是指定导入的命名空间,需要先创建

测试

nginx.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: prod
spec:
selector:
app: nginx
type: ClusterIP
ports:
- name: default
port: 80
protocol: TCP
targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx
namespace: prod
spec:
tls:
- hosts:
- nginx.nas.pocketdigi.com
secretName: nas-pocketdigi-com
rules:
- host: nginx.nas.pocketdigi.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: nginx
namespace: prod
spec:
progressDeadlineSeconds: 180
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
imagePullPolicy: Always
image: nginx:1.17.1-alpine
resources:
requests:
memory: "256Mi"
cpu: "0.1"
limits:
memory: "384Mi"
cpu: "0.8"
livenessProbe:
failureThreshold: 3
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 120
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 2
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 20
successThreshold: 2
timeoutSeconds: 3
1
kubectl apply -f nginx.yaml

如果nginx.nas.pocketdigi.com已经解析到这台主机,现在应该能正常访问到nginx。