使用国内资源部署 K3s RKE2 Rancher

国内不管是部署 K8s 还是部署其它 K8s 发行版，没有代理都是无法“正常”部署的，虽然可以为 CRI-O 和 Containerd 配置代理，但是手动部署 K8s 还是较为麻烦，这里记录一下 Rancher 使用国内资源进行快速部署的方法，但由于镜像更新不及时，实际部署过程中可能不会很顺利。

涉及到的国内资源站点：

• Rancher Release Mirrors: https://mirror.rancher.cn/ （由社区维护而非官方）
• 阿里云镜像仓库：registry.cn-hangzhou.aliyuncs.com

这些资源都由社区维护和上传，所以更新的频率不会很高，凑合着用吧。

K3s

使用国内资源进行部署：

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh |\
INSTALL_K3S_MIRROR=cn \
K3S_TOKEN=12345 sh -s - \
--system-default-registry=registry.cn-hangzhou.aliyuncs.com

部署过程中可以看到下载二进制和哈希文件是通过国内资源站点进行下载：

[INFO]  Finding release for channel stable
[INFO]  Using v1.30.5+k3s1 as release
[INFO]  Downloading hash rancher-mirror.rancher.cn/k3s/v1.30.5-k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary rancher-mirror.rancher.cn/k3s/v1.30.5-k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

所需的 image 均从 registry.cn-hangzhou.aliyuncs.com 中拉取：

root@k3s:~# kubectl get pod -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-d8b4bcd69-vkfkc                   1/1     Running     0          2m11s
kube-system   helm-install-traefik-crd-zz2tb            0/1     Completed   0          2m12s
kube-system   helm-install-traefik-rjg2h                0/1     Completed   1          2m12s
kube-system   local-path-provisioner-55bf9df6d7-jpv49   1/1     Running     0          2m11s
kube-system   metrics-server-7b58468669-8p7nw           1/1     Running     0          2m11s
kube-system   svclb-traefik-652949b2-s78sc              2/2     Running     0          80s
kube-system   traefik-69d6585798-khdp5                  1/1     Running     0          80s
root@k3s:~# crictl images
IMAGE                                                                TAG                    IMAGE ID            SIZE
registry.cn-hangzhou.aliyuncs.com/rancher/klipper-helm               v0.9.2-build20240828   1932cb543c3e4       72.2MB
registry.cn-hangzhou.aliyuncs.com/rancher/klipper-lb                 v0.4.9                 11a5d8a9f31aa       4.99MB
registry.cn-hangzhou.aliyuncs.com/rancher/local-path-provisioner     v0.0.28                5d221316a3c61       18.4MB
registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns   1.11.3                 c69fa2e9cbf5f       18.6MB
registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-library-traefik   2.11.8                 019c70fd40547       46.9MB
registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-metrics-server    v0.7.2                 48d9cfaaf3904       19.5MB
registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause             3.6                    6270bb605e12e       301kB

K3s agent 节点安装：

K3s agent 节点的安装不需要指定 mirror 地址了：

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | \
INSTALL_K3S_MIRROR=cn \
K3S_URL=https://x.x.x.x:6443 \
K3S_TOKEN=12345 \
sh -

高可用安装

前面的 K3s 部署属于单个 Master 节点部署方式，etcd 数据库无法实现高可用，高可用安装需要部署多台 Master 节点。

部署第一个 K3s Master 节点：

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh |\
INSTALL_K3S_MIRROR=cn \
K3S_TOKEN=12345 \
sh -s - server \
--cluster-init \
--token 12345 \
--system-default-registry=registry.cn-hangzhou.aliyuncs.com

随后将第二台，第三台 K3s Master 加入集群：

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh |\
INSTALL_K3S_MIRROR=cn \
K3S_TOKEN=12345 \
sh -s - server \
--server https://<first-master-ip>:6443 \
--system-default-registry=registry.cn-hangzhou.aliyuncs.com

通过配置文件的方式安装 K3s

在 K3s 节点上：

mkdir -p /etc/rancher/k3s
cat > /etc/rancher/k3s/config.yaml <<EOF
token: 12345
system-default-registry: registry.cn-hangzhou.aliyuncs.com
EOF

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh |\
INSTALL_K3S_MIRROR=cn sh -

K3s 进程启动后会自动读取 /etc/rancher/k3s/config.yaml 文件中的配置。

RKE2

设置部署使用国内资源的配置文件：

mkdir -p /etc/rancher/rke2/
cat > /etc/rancher/rke2/config.yaml <<EOF
token: 12345
system-default-registry: registry.cn-hangzhou.aliyuncs.com
EOF

下载并执行安装脚本：

curl -sfL https://rancher-mirror.rancher.cn/rke2/install.sh | \
INSTALL_RKE2_MIRROR=cn sh -

执行的结果：

[INFO]  finding release for channel stable
[INFO]  using v1.30.4-rke2r1 as release
[INFO]  downloading checksums at https://rancher-mirror.rancher.cn/rke2/releases/download/v1.30.4-rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://rancher-mirror.rancher.cn/rke2/releases/download/v1.30.4-rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local

随后启动服务：

systemctl start rke2-server.service

验证相关 image 已成功拉取：

export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
/var/lib/rancher/rke2/bin/crictl images

RKE2 agent 节点的安装

首先也需要配置 config.yaml ：

mkdir -p /etc/rancher/rke2/
cat > /etc/rancher/rke2/config.yaml <<EOF
server: https://<first-rke2-master-ip>:9345
token: 12345
EOF

随后通过安装脚本进行安装：

curl -sfL https://rancher-mirror.rancher.cn/rke2/install.sh | \
INSTALL_RKE2_MIRROR=cn \
INSTALL_RKE2_TYPE="agent" \
sh -

高可用安装

将第二台和第三台 RKE2 Master 加入集群：

mkdir -p /etc/rancher/rke2/
cat > /etc/rancher/rke2/config.yaml <<EOF
server: https://<first-rke2-master-ip>:9345
token: 12345

随后通过安装脚本进行安装：

curl -sfL https://rancher-mirror.rancher.cn/rke2/install.sh | \
INSTALL_RKE2_MIRROR=cn sh -

Rancher

通过 Rancher 可以方便部署和管理 K8s 集群。

高可用模式安装 Rancher

采用 Helm 的方式来安装 Rancher：

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo add jetstack https://charts.jetstack.io

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.0/cert-manager.crds.yaml

helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace

kubectl create ns cattle-system

helm install rancher rancher-latest/rancher \
--namespace cattle-system \
--set hostname=192.168.131.129.sslip.io \
--set replicas=1 \
--set bootstrapPassword=admin \
--set rancherImage=registry.cn-hangzhou.aliyuncs.com/rancher/rancher \
--set systemDefaultRegistry=registry.cn-hangzhou.aliyuncs.com

其中 helm repo add 添加 repo 还需使用代理
其次使用 helm 部署 cert-manager 时，需要配置 containerd 访问 docker.io 的 mirror 否则国内无法拉取所需的映像（参考后面——配置 Mirror）。

登录 Rancher 验证 global 设置：

单节点安装 Rancher

单节点可以使用 Docker 的方式运行 Rancher 容器：

docker run -d --restart=unless-stopped \
-p 80:80 -p 443:443 \
--privileged \
-e CATTLE_SYSTEM_DEFAULT_REGISTRY=registry.cn-hangzhou.aliyuncs.com \
--name rancher \
registry.cn-hangzhou.aliyuncs.com/rancher/rancher:v2.9.2

配置 Mirror

为 K3s 和 RKE2 配置 Mirror 不能直接更改 containerd 的配置文件，而是修改相关 K3s 或 RKE2 的配置文件，让其自动生成。

K3s

配置 Mirror：

cat >> /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  "docker.io":
    endpoint:
      - "https://registry.cn-hangzhou.aliyuncs.com"
EOF

systemctl restart k3s

重启后验证配置：

cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]

[host]
[host."https://registry.cn-hangzhou.aliyuncs.com/v2"]
  capabilities = ["pull", "resolve"]

RKE2

RKE2 mirror 的配置与 K3s 类似：

cat >> /etc/rancher/rke2/registries.yaml <<EOF
mirrors:
  "docker.io":
    endpoint:
      - "https://registry.cn-hangzhou.aliyuncs.com"
EOF
systemctl restart rke2-server

重启后验证配置：

cat /var/lib/rancher/rke2/agent/etc/containerd/certs.d/docker.io/hosts.toml
# File generated by rke2. DO NOT EDIT.

server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]

[host]
[host."https://registry.cn-hangzhou.aliyuncs.com/v2"]
  capabilities = ["pull", "resolve"]