Kubernetes部署

第一部分：理解 K8s 原理

1.1 应用部署发展过程

1. 传统物理机部署：软件直接装在服务器。缺点：资源没法限制，一个程序吃满 CPU 内存，会把别的程序搞崩。
2. 虚拟机部署：一台物理机跑多台虚拟机，每台虚拟机独立操作系统。缺点：每台虚拟机完整装一套系统，特别吃内存硬盘，资源浪费大。
3. 容器部署 (Docker)：多个容器共享宿主机操作系统内核，每个容器打包自己的程序和依赖，体积小、启动快。

问题来了：机器少手动启动容器还行；几十上百台服务器，成千上万个容器，手动管不过来：容器挂了怎么自动重启？访问量变大怎么自动多开容器？怎么把请求分发到多个容器？ → 这就需要容器编排工具 K8s，专门帮我们批量管理容器。

答：这就需要容器编排工具 K8s，专门帮我们批量管理容器。

1.2 K8s 集群两大角色（控制节点 + 工作节点）

整个 K8s 集群分成两类机器：

Master 控制节点（做决策，不跑业务程序）

1. apiserver：整个集群唯一入口，所有操作（创建 / 删除应用）都找它，做身份校验，读写集群数据。
2. etcd：集群数据库，存全部集群信息：有多少机器、有多少容器、配置是什么。
3. scheduler 调度器：新的容器该放到哪台 worker 机器上跑，它来选机器。
4. controller‑manager 控制器：盯着集群状态，如果容器挂了，就命令重新拉起，保证符合你的预期。

Node 工作节点（干活，真正跑业务容器）

1. kubelet：每台 worker 机器上的代理，接收 master 大脑下发指令，管理本机容器生命周期。
2. 容器运行时：真正创建、启动容器的软件，本实验用 Docker，因为 k8s1.24 之后不再原生支持 docker，必须装cri‑dockerd适配器做中转。
3. kube‑proxy：网络代理，负责服务发现、负载均衡，外部访问业务的时候转发流量。

最小单元叫Pod：Pod 是 k8s 最小单元，一个 Pod 里面可以放 1 个或者多个容器，同一个 Pod 容器共享网络、共享存储。平时我们部署应用本质是部署 Pod。

1.3 为什么我们要部署 Harbor 私有镜像仓库？

镜像就是容器安装包。默认镜像要去外网下载，实验室环境外网慢甚至不通。

Harbor 就是本地局域网的镜像仓库服务器：把所有 k8s 需要的组件镜像下载下来传到 harbor，所有 k8s 机器从内网拉镜像，速度飞快。

1.4 为什么要用 cri‑dockerd？

K8s 1.24 版本开始官方抛弃 dockershim，不再原生对接 Docker。

Docker 本身不能直接被 k8s 调用。cri‑dockerd就是一个 “翻译适配器”：k8s 发标准 CRI 接口指令，适配器转成 Docker 能听懂的指令，让 Docker 继续作为容器运行时。

1.5 网络插件 Flannel 是干嘛的？

刚初始化完 k8s 集群，节点状态是NotReady没准备好。

因为 k8s 需要一套网络：不同机器上 Pod 之间要互相通信。Flannel 就是网络插件，搭建集群内部网络，装好之后节点状态变成 Ready。

1.6 必做环境预处理，为什么？

1. 关闭 swap 交换分区：swap 是硬盘虚拟内存。k8s 不允许开启 swap，内存不够宁愿把容器杀掉也不用硬盘虚拟内存，会影响调度性能。
2. 内核参数开启网桥转发：容器之间网络互通，需要开启内核 iptables 转发。
3. 主机解析 /etc/hosts：各个机器通过域名能互相找到，不用记复杂 IP。

第二部分：完整实验部署

集群规划：

步骤一：部署 Harbor 服务器

1.1 制作本地 docker yum 源

[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name = docker
baseurl = https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/
gpgcheck = 0
EOF

[root@harbor ~]# dnf install httpd createrepo -y
[root@harbor ~]# mkdir /var/www/html/docker/ -p
[root@harbor ~]# vim /etc/httpd/conf/httpd.conf

[root@harbor ~]# systemctl enable --now httpd

[root@harbor ~]# dnf install docker-ce --downloadonly --destdir /mnt/ -y
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
[root@harbor ~]# createrepo -v /var/www/html/docker/

[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name = docker
baseurl = http://172.25.254.200:4444/docker
gpgcheck = 0
EOF

1.2 Harbor 节点配置 docker 和内核参数

[root@harbor ~]# dnf install docker-ce -y
[root@harbor ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@harbor ~]# modprobe -a br_netfilter
[root@harbor ~]# vim /etc/sysctl.d/docker.conf

[root@harbor ~]# systemctl restart systemd-modules-load.service
[root@harbor ~]# sysctl --system

[root@harbor ~]# vim /lib/systemd/system/docker.service

[root@harbor ~]# systemctl daemon-reload
[root@harbor ~]# systemctl enable --now docker

1.3 生成自签名证书

[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# openssl req -newkey rsa:4096 \
-nodes -sha256 -keyout /data/certs/timinglee.org.key \
-addext "subjectAltName = DNS:reg.timinglee.org" \
-x509 -days 365 -out /data/certs/timinglee.org.crt

1.4 解压配置 Harbor

[root@harbor ~]# tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/
[root@harbor ~]# cd /opt/harbor/

[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml

[root@harbor harbor]# ./install.sh --with-chartmuseum

1.5 设置 harbor 为 systemd 服务管理

[root@harbor harbor]# vim /lib/systemd/system/harbor.service

[root@harbor harbor]# systemctl daemon-reload
[root@harbor harbor]# systemctl enable --now harbor
[root@harbor harbor]# docker compose up
[root@harbor harbor]# docker compose down

1.6 docker 信任 harbor 证书

[root@harbor harbor]# mkdir /etc/docker/certs.d/reg.timinglee.org/ -p
[root@harbor harbor]# cp /data/certs/timinglee.org.crt /etc/docker/certs.d/reg.timinglee.org/ca.crt
[root@harbor harbor]# vim /etc/hosts

[root@harbor harbor]# systemctl restart docker

步骤二：所有 k8s 节点统一环境配置

2.1 所有主机彻底关闭 swap 分区

以harbor为例：

[root@harbor harbor]# sed '/swap/s/^/#/g' -i /etc/fstab
[root@harbor harbor]# reboot

2.2 所有节点配置 /etc/hosts 主机解析

[root@harbor ~]# vim /etc/hosts

[root@harbor ~]# for i in 150 30 40
> do
> scp /etc/hosts root@172.25.254.$i:/etc/hosts
> done

步骤三：利用Ansible部署kubernetes运行环境

3.1 安装ansible

[root@harbor ~]# vim /etc/yum.repos.d/epel.repo

[root@harbor ~]# dnf install ansible-core.x86_64 ansible.noarch -y

3.2 配置所需用户

[root@harbor ~]# useradd devops
[root@harbor ~]# echo lee | passwd --stdin devops
[root@harbor ~]# su - devops
[devops@harbor ~]$ mkdir ansible
[devops@harbor ~]$ cd ansible/
[devops@harbor ansible]$

3.3 生成ansible的临时配置文件并建立运行环境

[devops@harbor ansible]$ vim inventory

[devops@harbor ansible]$ vim ansible.cfg

#建立用户
[devops@harbor ansible]$ ansible all -m user -a "name=devops state=present"
[devops@harbor ansible]$ ansible all -m shell -a 'echo "lee" | passwd --stdin devops'
[devops@harbor ansible]$ ansible all -m lineinfile -a 'path=/etc/sudoers line="devops ALL=(ALL) NOPASSWD: ALL" insertafter=EOF'

3.4生成ansible的主配置文件并测试

[devops@harbor ansible]$ vim ansible.cfg

测试：

3.5 利用ansible剧本部署docker

#关闭防火墙

[root@harbor ~]# systemctl disable --now firewalld.service
Removed "/etc/systemd/system/multi-user.target.wants/firewalld.service".
Removed "/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service".

[devops@harbor ansible]$ su -
密码：
[root@harbor ~]# cp /opt/harbor/common/config/registry/root.crt /home/devops/ansible/ca.crt
[root@harbor ~]# chown devops:devops /home/devops/ansible/ca.crt
[root@harbor ~]# exit
注销
[devops@harbor ansible]$ ls -l /home/devops/ansible/ca.crt
-rw-r--r-- 1 devops devops 0 8月 22 19:55 /home/devops/ansible/ca.crt

[devops@harbor ansible]$ vim kubernetes-devops.yml
- name: devops k8s from ansible
hosts: all
become: true
gather_facts: true
tasks:
- name: setup docker repo
yum_repository:
name: docker
description: docker
baseurl: http://172.25.254.200:4444/docker
file: docker
gpgcheck: no

- name: install docker
dnf:
name: docker-ce
state: present

- name: modify docker.service ExecStart
replace:
path: /lib/systemd/system/docker.service
regexp: '^(ExecStart=/usr/bin/dockerd.*)$'
replace: 'ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true'
backup: yes

- name: systemctl daemon‑reload after docker.service change
systemd:
daemon_reload: yes

- name: setup docker registry daemon.json
copy:
content: |
{
"registry-mirrors":["https://reg.timinglee.org"]
}
dest: /etc/docker/daemon.json
mode: '0644'

- name: create docker cert directory
file:
path: /etc/docker/certs.d/reg.timinglee.org
state: directory
mode: '0755'

- name: copy harbor ca certificate
copy:
src: "./ca.crt"
dest: "{{ item.dest }}"
loop:
- { dest: "/etc/docker/certs.d/reg.timinglee.org/ca.crt" }
- { dest: "/etc/pki/ca-trust/source/anchors/ca.crt" }

- name: load kernel module br_netfilter config
copy:
dest: "{{ item.dest }}"
content: "{{ item.value }}"
loop:
- dest: /etc/modules-load.d/docker_mod.conf
value: br_netfilter
- dest: /etc/sysctl.d/docker.conf
value: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

- name: load br_netfilter kernel module
community.general.modprobe:
name: br_netfilter
state: present

- name: refresh ca trust & reload sysctl
shell: update-ca-trust extract; sysctl --system
changed_when: false

- name: start and enable services
service:
name: "{{ item }}"
state: restarted
enabled: yes
loop:
- docker
- systemd-modules-load.service

[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml

3.6 利用ansible安装cri-dockerd

[devops@harbor ansible]$ tar zxf cri-dockerd-0.4.4.amd64.tgz

[devops@harbor ansible]$ cd cri-dockerd/

[devops@harbor cri-dockerd]$ vim cri-docker.service

[devops@harbor ansible]$ vim kubernetes-devops.yml
- name: devops k8s from ansible
hosts: all
tasks:
- name: setup docker repo
yum_repository:
name: docker
description: docker
baseurl: http://172.25.254.200:4444/docker
file: docker
gpgcheck: no

- name: install docker
dnf:
name: docker-ce
state: present

- name: setup docker.service
replace:
path: /lib/systemd/system/docker.service
regexp: "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock"
replace: "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true"

- name: setup docker registry
copy:
content: |
{
"registry-mirrors":["https://reg.timinglee.org"]
}
dest: /etc/docker/daemon.json

- name: setup docker certs
file:
path: /etc/docker/certs.d/reg.timinglee.org
state: directory

- name: cp certs file
copy:
src: "{{item.src}}"
dest: "{{item.dest}}"
loop:
- src: /etc/docker/certs.d/reg.timinglee.org/ca.crt
dest: /etc/docker/certs.d/reg.timinglee.org/ca.crt

- src: /etc/docker/certs.d/reg.timinglee.org/ca.crt
dest: /etc/pki/ca-trust/source/anchors/ca.crt

########################################################
- name: stup cri-dockerd
copy:
src: "{{item.src}}"
dest: "{{item.dest}}"
mode: '0755'
loop:
- src: ./cri-dockerd/cri-dockerd
dest: /usr/bin/cri-dockerd

- src: "./cri-dockerd/cri-docker.service"
dest: "/lib/systemd/system/cri-docker.service"

- src: "./cri-dockerd/cri-docker.socket"
dest: "/lib/systemd/system/cri-docker.socket"

- name: update ca cert
shell: "update-ca-trust extract;sysctl --system;systemctl daemon-reload"

- name: start services
service:
name: "{{item}}"
state: restarted
enabled: yes
loop:
- docker
- systemd-modules-load.service
- cri-docker

[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml

3.7 利用ansible部署k8s软件

[devops@harbor ansible]$ mv kubernetes-devops.yml docker-devops.yml

[devops@harbor ansible]$ vim kubernetes-devops.yml
- name: install kubernetes from ansible
hosts: all
tasks:
- name: setup k8s repo
yum_repository:
name: kubernetes
description: kubernetes
baseurl: https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm
file: kubernetes
gpgcheck: no

- name: install software in master
dnf:
name: "{{item}}"
state: present
loop:
- kubeadm-1.35.7-150500.1.1
- kubelet-1.35.7-150500.1.1
- kubectl-1.35.7-150500.1.1
when: inventory_hostname == "172.25.254.150"

- name: install software in work node
dnf:
name: "{{item}}"
state: present
loop:
- kubeadm-1.35.7-150500.1.1
- kubelet-1.35.7-150500.1.1
when: inventory_hostname == "172.25.254.30" or inventory_hostname == "172.25.254.40"

- name: setup kubectl kubeadm complication
lineinfile:
path: /root/.bashrc
line: |
source <(kubectl completion bash)
source <(kubeadm completion bash)
when: inventory_hostname == "172.25.254.150"

- name: start kubelet
service:
name: kubelet
state: started
enabled: yes

[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml

#让补齐功能生效
[root@master ~]# source ~/.bashrc
[root@master ~]# kubectl 命令后按2此tab键
alpha (Commands for features in alpha)
annotate (更新一个资源的注解)
api-resources (Print the supported API resources on the server)
api-versions (Print the supported API versions on the server, in the form of "group/version")
apply (Apply a configuration to a resource by file name or stdin)
attach (挂接到一个运行中的容器)
auth (Inspect authorization)
autoscale (Auto-scale a deployment, replica set, stateful set, or replication controller)
certificate (Modify certificate resources)

[root@master ~]# kubeadm