KubeVirt 支持虚拟机快照和克隆，但是都依赖 Kubernetes 中 CSI 提供的功能。
快照 在做快照之前需要创建 VolumeSnapshot 资源，参考以下连接：
安装好 snapshot controller 和创建好 VolumeSnapshotClass 后，可以通过以下命令进行检查：
1 2 3 vagrant@master01:~$ kubectl get volumesnapshotclass NAME DRIVER DELETIONPOLICY AGE csi-rbdplugin-snapclass rook-ceph.rbd.csi.ceph.com Delete 10s
启用快照的特性 同样的， KubeVirt 的快照功能也是需要手动开启的，可以直接编辑 kubevirt 进行开启：
1 2 3 4 5 6 7 8 9 vagrant@master01:~$ kubectl edit kubevirt kubevirt -n kubevirt spec: certificateRotateStrategy: {} configuration: developerConfiguration: featureGates: - HotplugVolumes - Snapshot
创建快照 KubeVirt 支持对在线和离线的虚拟机创建快照。qemu agent guest 处于运行的状态，确保在拍摄快照的时候冻结文件系统，拍摄完成后解除冻结，如果虚拟机没有运行 qemu agent guest 将无法拍摄运行时的快照，只能关机拍摄快照。
查看虚拟机中的 qemu agent guest 的状态，只需查看 VM 中的 AgentConnected ：
1 2 3 vagrant@master01:~$ k describe vm fedora-vm -n cdi-test | grep -A 1 Agent Type: AgentConnected Created: true
使用以下资源定义为 cdi-test 命令空间中的 fedora-vm 创建快照：
1 2 3 4 5 6 7 8 9 apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineSnapshot metadata: name: snap-fedora-vm spec: source : apiGroup: kubevirt.io kind: VirtualMachine name: fedora-vm
拍摄快照：
1 2 3 4 5 6 7 8 9 10 11 12 13 vagrant@master01:~$ kubectl get vmi -n cdi-test NAME AGE PHASE IP NODENAME READY fedora-vm 40m Running 10.244.241.93 master01 True vagrant@master01:~$ kubectl -n cdi-test apply -f snap-fedora-vm.yaml virtualmachinesnapshot.snapshot.kubevirt.io/snap-fedora-vm created vagrant@master01:~$ kubectl wait vmsnapshot snap-fedora-vm -n cdi-test --for condition=Ready virtualmachinesnapshot.snapshot.kubevirt.io/snap-fedora-vm condition met vagrant@master01:~$ kubectl get volumesnapshot -n cdi-test NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE vmsnapshot-3331182f-bd76-4f9b-9e93-c253322c1f49-volume-disk0 true fedora 5Gi csi-rbdplugin-snapclass snapcontent-77ea2f79-0411-4702-b343-1c93112a6142 17s 18s vmsnapshot-3331182f-bd76-4f9b-9e93-c253322c1f49-volume-example-volume-hotplug true example-volume-hotplug 10Gi csi-rbdplugin-snapclass snapcontent-62f50a95-6987-44bb-b248-74988a0c7b70 17s 18s
可以看到针对 fedora-vm 创建了两个快照，因为前面附加了一块磁盘。
恢复快照 在恢复快照之前，对 fedora-vm 进行内容的更改，方便恢复快照后进行对比：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa fedora@fedora-vm.cdi-test Last login: Sun Apr 28 10:41:35 2024 from 10.244.30.110 [fedora@fedora-vm ~]$ sudo -i [root@fedora-vm ~]# echo test > /etc/test.txt [root@fedora-vm ~]# parted /dev/sda mklabel msdos Information: You may need to update /etc/fstab. [root@fedora-vm ~]# parted /dev/sda mkpart primary xfs 2048s 100% Information: You may need to update /etc/fstab. [root@fedora-vm ~]# udevadm settle [root@fedora-vm ~]# lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 10G 0 disk └─sda1 8:1 0 10G 0 part
首先需要将 VM 停止：
1 2 3 4 5 6 vagrant@master01:~$ virtctl stop fedora-vm -n cdi-test VM fedora-vm was scheduled to stop vagrant@master01:~$ kubectl get vm -n cdi-test NAME AGE STATUS READY fedora-vm 24h Stopped False
然后使用以下资源定义进行还原快照：
1 2 3 4 5 6 7 8 9 10 apiVersion: snapshot.kubevirt.io/v1alpha1 kind: VirtualMachineRestore metadata: name: restore-fedora-vm spec: target: apiGroup: kubevirt.io kind: VirtualMachine name: fedora-vm virtualMachineSnapshotName: snap-fedora-vm
还原快照：
1 2 3 4 5 6 vagrant@master01:~$ kubectl -n cdi-test apply -f restore-fedora-vm.yaml virtualmachinerestore.snapshot.kubevirt.io/restore-fedora-vm created vagrant@master01:~$ kubectl get virtualmachinerestore -n cdi-test NAME TARGETKIND TARGETNAME COMPLETE RESTORETIME ERROR restore-fedora-vm VirtualMachine fedora-vm true 41s
启动 VM 进行检查磁盘是否被还原：
1 2 3 4 5 6 7 8 9 10 vagrant@master01:~$ virtctl start fedora-vm -n cdi-test VM fedora-vm was scheduled to start vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa fedora@fedora-vm.cdi-test Last login: Sun Apr 28 10:41:35 2024 from 10.244.30.110 [fedora@fedora-vm ~]$ cat /etc/test.txt cat : /etc/test.txt: No such file or directory[fedora@fedora-vm ~]$ lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 10G 0 disk
可以看到两块的磁盘的内容已经还原到了拍摄快照时候的内容。
克隆 在满足 Snapshot & Restore 后，就可以对虚拟机实现克隆了，如果不满足，参考前面的链接首先创建 VolumeSnapshotClass 。
通过创建 VirtualMachineClone CDR 资源来对虚拟机进行克隆：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 kind: VirtualMachineClone apiVersion: "clone.kubevirt.io/v1alpha1" metadata: name: testclone spec: source : apiGroup: kubevirt.io kind: VirtualMachine name: fedora-vm target: apiGroup: kubevirt.io kind: VirtualMachine name: fedora-vm-clone labelFilters: - "*" - "!someKey/*" annotationFilters: - "anotherKey/*" template: labelFilters: - "*" - "!someKey/*" annotationFilters: - "anotherKey/*" newMacAddresses: interfaceName: "00-11-22" newSMBiosSerial: "new-serial"
其中 labelFilters 和 annotationFilters 是筛选要 copy 的标签和注释。template 一些网络 CNI 会将网络信息注入到 template.label 或 template.annotation 中，此处也是筛选要 copy 的标签和注释。newMacAddresses 是在克隆时为特定的网卡设置新的 Mac Addresses。newSMBiosSerial 也是一样克隆时生成新的 SMBios Serial。
克隆虚拟机：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 vagrant@master01:~$ cat testclone.yaml kind: VirtualMachineClone apiVersion: "clone.kubevirt.io/v1alpha1" metadata: name: testclone spec: source : apiGroup: kubevirt.io kind: VirtualMachine name: fedora-vm target: apiGroup: kubevirt.io kind: VirtualMachine name: fedora-vm-clone labelFilters: - "*" annotationFilters: - "*" template: labelFilters: - "*" annotationFilters: - "*" vagrant@master01:~$ kubectl -n cdi-test apply -f testclone.yaml virtualmachineclone.clone.kubevirt.io/testclone created
验证克隆的虚拟机：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 vagrant@master01:~$ kubectl get virtualmachineclone -n cdi-test NAME PHASE SOURCEVIRTUALMACHINE TARGETVIRTUALMACHINE testclone Succeeded fedora-vm fedora-vm-clone vagrant@master01:~$ kubectl get vm -n cdi-test NAME AGE STATUS READY fedora-vm 25h Running True fedora-vm-clone 51s Stopped False vagrant@master01:~$ virtctl start fedora-vm-clone -n cdi-test VM fedora-vm-clone was scheduled to start vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa fedora@fedora-vm-clone.cdi-test The authenticity of host 'vmi/fedora-vm-clone.cdi-test:22 (192.168.121.1:7890)' can't be established. ECDSA key fingerprint is SHA256:xyti2s597SmKOZ+m4Jalsgi5CdW65CvlzjgvasU/v6s. Are you sure you want to continue connecting (yes/no)? yes Last login: Mon Apr 29 06:43:51 2024 from 10.244.30.66 [fedora@fedora-vm-clone ~]$ [fedora@fedora-vm-clone ~]$ hostnamectl status Static hostname: fedora-vm-clone Icon name: computer-vm Chassis: vm 🖴 Machine ID: bb0bebea15a455d18a61c5e10c294b1d Boot ID: 68efaf1d0f2149a8811ed34e7e6909df Virtualization: qemu Operating System: Fedora Linux 37 (Cloud Edition) CPE OS Name: cpe:/o:fedoraproject:fedora:37 Kernel: Linux 6.0.7-301.fc37.x86_64 Architecture: x86-64 Hardware Vendor: KubeVirt Hardware Model: None Firmware Version: 1.16.1-1.el9
虚拟机封装 当从官网下载构建好的虚拟机后，在企业中针对每个虚拟机还要做好安全基线，监控等任务，如果每个虚拟机都手动一遍一遍从头操作的话，效率很低下，比较好的方法是制作虚拟机的模板，然后基于模板创建虚拟机。
基于现有的虚拟机做模板首先需要对虚拟机进行封装（sysprep），封装的过程会清楚主机特有的信息。
先假设对默认的虚拟机做设置：
1 2 3 4 5 6 7 vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa root@centos9s-vm.vm-clone The authenticity of host 'vmi/centos9s-vm.vm-clone:22 (192.168.121.168:6443)' can't be established. ECDSA key fingerprint is SHA256:rMimCVGTHp59dJFFJRfEs8ItM67IsmuR4CCOyq1zCFg. Are you sure you want to continue connecting (yes/no)? yes Activate the web console with: systemctl enable --now cockpit.socket [root@centos9s-vm ~]# echo "CentOS Stream 9 Template" > /etc/motd
在对虚拟机封装之前，首先要停止虚拟机：
1 2 3 4 5 6 vagrant@master01:~$ virtctl stop centos9s-vm -n vm-clone VM centos9s-vm was scheduled to stop vagrant@master01:~$ kubectl get vm -n vm-clone NAME AGE STATUS READY centos9s-vm 9m14s Stopped False
使用 virtctl guestfs 对虚拟机使用的 PVC 进行挂载：
1 2 3 4 5 6 7 vagrant@master01:~$ virtctl guestfs centos9s -n vm-clone Use image: quay.io/kubevirt/libguestfs-tools:v1.2.0 The PVC has been mounted at /dev/vda Waiting for container libguestfs still in pending, reason: ContainerCreating, message: If you don't see a command prompt, try pressing enter. bash-5.1$
此时 PVC 已经挂载到了 /dev/vda ，接下来对 vda 进行封装：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 bash-5.1$ virt-sysprep -a /dev/vda [ 0.0] Examining the guest ... [ 6.5] Performing "abrt-data" ... [ 6.6] Performing "backup-files" ... [ 7.6] Performing "bash-history" ... [ 7.6] Performing "blkid-tab" ... [ 7.7] Performing "crash-data" ... [ 7.7] Performing "cron-spool" ... [ 7.7] Performing "dhcp-client-state" ... [ 7.7] Performing "dhcp-server-state" ... [ 7.7] Performing "dovecot-data" ... [ 7.8] Performing "ipa-client" ... [ 7.8] Performing "kerberos-hostkeytab" ... [ 7.8] Performing "logfiles" ... [ 7.9] Performing "lvm-system-devices" ... [ 7.9] Performing "machine-id" ... [ 8.0] Performing "mail-spool" ... [ 8.0] Performing "net-hostname" ... [ 8.0] Performing "net-hwaddr" ... [ 8.1] Performing "net-nmconn" ... [ 8.1] Performing "pacct-log" ... [ 8.1] Performing "package-manager-cache" ... [ 8.2] Performing "pam-data" ... [ 8.2] Performing "passwd-backups" ... [ 8.2] Performing "puppet-data-log" ... [ 8.2] Performing "rh-subscription-manager" ... [ 8.2] Performing "rhn-systemid" ... [ 8.3] Performing "rpm-db" ... [ 8.3] Performing "samba-db-log" ... [ 8.3] Performing "script" ... [ 8.3] Performing "smolt-uuid" ... [ 8.3] Performing "ssh-hostkeys" ... [ 8.3] Performing "ssh-userdir" ... [ 8.4] Performing "sssd-db-log" ... [ 8.4] Performing "tmp-files" ... [ 8.4] Performing "udev-persistent-net" ... [ 8.4] Performing "utmp" ... [ 8.4] Performing "yum-uuid" ... [ 8.4] Performing "customize" ... [ 8.4] Setting a random seed [ 8.5] Setting the machine ID in /etc/machine-id [ 8.5] SELinux relabelling [ 27.8] Performing "lvm-uuids" ...
执行完毕后 exit 退出即可，此时不要启动 VM，否则又会生成主机特有信息。
如果虚拟机使用的文件系统是 Btrfs ，将会遇到错误（所以我使用了 CentOS9Stream 作为测试）：
1 2 3 4 5 6 7 8 9 bash-5.1$ virt-sysprep -a /dev/vda [ 0.0] Examining the guest ... virt-sysprep: error: libguestfs error: inspect_os: mount exited with status 32: mount: /tmp/btrfse339EF: unknown filesystem type 'btrfs' . If reporting bugs, run virt-sysprep with debugging enabled and include the complete output: virt-sysprep -v -x [...]
通过使用 DataVolume 对虚拟机的根盘的克隆：
1 2 3 4 5 6 7 8 9 10 11 12 13 apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: "centos9s-template" spec: storage: resources: requests: storage: 30Gi source : pvc: name: centos9s namespace: vm-clone
应用：
1 2 3 4 5 6 7 vagrant@master01:~$ kubectl -n vm-clone apply -f dv_template.yaml datavolume.cdi.kubevirt.io/centos9s-template created vagrant@master01:~$ vagrant@master01:~$ kubectl get dv -n vm-clone NAME PHASE PROGRESS RESTARTS AGE centos9s Succeeded 100.0% 3h34m centos9s-template Succeeded 100.0% 9s
后续可以基于 centos9s-template 再做克隆卷，然后用于虚拟机。
1 2 3 4 5 vagrant@master01:~$ kubectl get dv -n vm-clone NAME PHASE PROGRESS RESTARTS AGE centos9s Succeeded 100.0% 3h37m centos9s-template Succeeded 100.0% 3m18s centos9s-template-vm1 Succeeded 100.0% 23s
基于 centos9s-template 创建了 centos9s-template-vm1 ，然后将 centos-template-vm1 用于 centos9s-template-vm1 虚拟机（此处仅仅是快速演示，最佳实践是创建 Virtual Machine Template 这个需要基于 OpenShift 的 Template 资源）：
1 2 3 4 5 6 7 8 9 10 11 vagrant@master01:~$ kubectl get vm -n vm-clone NAME AGE STATUS READY centos9s-template-vm1 13s Running True centos9s-vm 22m Stopped False vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa root@centos9s-template-vm1.vm-clone The authenticity of host 'vmi/centos9s-template-vm1.vm-clone:22 (192.168.121.168:6443)' can't be established. ECDSA key fingerprint is SHA256:l+naPMcmcDr+zslu4Uqakpuxq9Mbfa1XwzbmObYkPRs. Are you sure you want to continue connecting (yes/no)? yes CentOS Stream 9 Template Activate the web console with: systemctl enable --now cockpit.socket
可以看到之前写入 motd 的内容了。
评论· · · · · ·