对象存储通过暴露 S3 API 以便应用放置和获取存储集群中的数据。
配置对象存储 Rook 能够在 Kubernetes 中部署对象存储或者连接到外部的 RGW 服务。
创建本地对象网关 部署本地 rgw 可以借助 CephObjectStore CRD，示例资源定义：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 apiVersion: ceph.rook.io/v1 kind: CephObjectStore metadata: name: my-store namespace: rook-ceph spec: metadataPool: failureDomain: host replicated: size: 3 dataPool: failureDomain: host erasureCoded: dataChunks: 2 codingChunks: 1 preservePoolsOnDelete: true gateway: sslCertificateRef: port: 80 instances: 1
应用并检查：
1 2 3 4 5 [vagrant@master01 rook-ceph]$ kubectl apply -f object.yaml cephobjectstore.ceph.rook.io/my-store created [vagrant@master01 rook-ceph]$ kubectl get pod -n rook-ceph -l app=rook-ceph-rgw NAME READY STATUS RESTARTS AGE rook-ceph-rgw-my-store-a-79ffd75dc8-fp95c 2/2 Running 0 87s
在 toolbox 中，验证 rgw 已部署：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 [vagrant@master01 rook-ceph]$ kubectl exec -it rook-ceph-tools-66b77b8df5-x97q4 -n rook-ceph -- /bin/ceph -s cluster: id : f8bdb7b9-12c8-4814-b4a2-6122366ddd1a health: HEALTH_WARN too many PGs per OSD (254 > max 250) services: mon: 3 daemons, quorum a,b,c (age 13m) mgr: a(active, since 12m), standbys: b mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 13m), 3 in (since 4w) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 15 pools, 265 pgs objects: 279 objects, 565 KiB usage: 112 MiB used, 60 GiB / 60 GiB avail pgs: 265 active+clean io: client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
验证相关的 Pool 已创建：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [vagrant@master01 rook-ceph]$ kubectl exec -it rook-ceph-tools-66b77b8df5-x97q4 -n rook-ceph -- /bin/ceph osd pool ls .mgr replicapool ecpool replicated-metadata-pool ec-data-pool myfs-metadata myfs-replicated my-store.rgw.control my-store.rgw.meta my-store.rgw.log my-store.rgw.buckets.index my-store.rgw.buckets.non-ec my-store.rgw.otp .rgw.root my-store.rgw.buckets.data
创建 Bucket Object Store 已经配置好了，下一步就是创建 bucket 让客户端能够读写对象。
storageclass-bucket.yaml 的示例定义：
1 2 3 4 5 6 7 8 9 10 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-ceph-bucket provisioner: rook-ceph.ceph.rook.io/bucket reclaimPolicy: Delete parameters: objectStoreName: my-store objectStoreNamespace: rook-ceph
应用并检查：
1 2 3 4 5 6 7 8 vagrant@master01 rook-ceph]$ kubectl apply -f storageclass-bucket.yaml storageclass.storage.k8s.io/rook-ceph-bucket created [vagrant@master01 rook-ceph]$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 20h rook-ceph-block-ec rook-ceph.rbd.csi.ceph.com Delete Immediate true 20h rook-ceph-bucket rook-ceph.ceph.rook.io/bucket Delete Immediate false 3s rook-cephfs rook-ceph.cephfs.csi.ceph.com Delete Immediate false 13h
基于创建出来的 storageclass，客户端可以请求创建 Object Bucket Claim(OBC)，这种资源类似于 PVC。
创建 OBC 的示例定义：
1 2 3 4 5 6 7 8 apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: ceph-bucket namespace: obc-demo spec: generateBucketName: ceph-bkt storageClassName: rook-ceph-bucket
应用并检查：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 [vagrant@master01 rook-ceph]$ kubectl apply -f object-bucket-claim.yaml objectbucketclaim.objectbucket.io/ceph-bucket created [vagrant@master01 rook-ceph]$ kubectl get obc -n obc-demo NAME AGE ceph-bucket 9s [vagrant@master01 rook-ceph]$ kubectl describe obc/ceph-bucket -n obc-demo Name: ceph-bucket Namespace: obc-demo Labels: bucket-provisioner=rook-ceph.ceph.rook.io-bucket Annotations: <none> API Version: objectbucket.io/v1alpha1 Kind: ObjectBucketClaim Metadata: Creation Timestamp: 2024-03-04T01:48:17Z Finalizers: objectbucket.io/finalizer Generation: 4 Resource Version: 181929 UID: c47194f4-8ab1-48c1-b436-2ac6c8313eb4 Spec: Bucket Name: ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9 Generate Bucket Name: ceph-bkt Object Bucket Name: obc-obc-demo-ceph-bucket Storage Class Name: rook-ceph-bucket Status: Phase: Bound Events: <none>
使用对象存储 应用需要使用对象存储需要以下信息：
RGW 的端点
RGW 的端口
ACCESS_KEY
SECRET_KEY
Bucket 名称
其中 Bucket 名称，可以查看创建的出来的 OBC 资源，前面创建的 ceph-bucket obc 所对应的 Bucket 是 ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9 。
随着 OBC 的创建，还会生成相应的 configmap 和 secret：
1 2 3 4 5 6 7 [vagrant@master01 rook-ceph]$ kubectl get secret -n obc-demo NAME TYPE DATA AGE ceph-bucket Opaque 2 2m26s [vagrant@master01 rook-ceph]$ kubectl get cm -n obc-demo NAME DATA AGE ceph-bucket 5 2m31s kube-root-ca.crt 1 2m35s
其中 secret 里面存储是 ACCESS_KEY 和 SECRET_KEY：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 [vagrant@master01 rook-ceph]$ kubectl get secret -o yaml ceph-bucket -n obc-demo apiVersion: v1 data: AWS_ACCESS_KEY_ID: UURVUlRDTEQxU0YxTE1MVzFWNUs= AWS_SECRET_ACCESS_KEY: MngwNGtQODI4ZUs2bXk3djFnSUJRelA5b0ZadHRSVG5mOWduazBRcQ== kind: Secret metadata: creationTimestamp: "2024-03-04T01:48:19Z" finalizers: - objectbucket.io/finalizer labels: bucket-provisioner: rook-ceph.ceph.rook.io-bucket name: ceph-bucket namespace: obc-demo ownerReferences: - apiVersion: objectbucket.io/v1alpha1 blockOwnerDeletion: true controller: true kind: ObjectBucketClaim name: ceph-bucket uid: c47194f4-8ab1-48c1-b436-2ac6c8313eb4 resourceVersion: "181924" uid: 295cd046-9ab3-4126-8112-1b2d09f4beef type : Opaque
configmap 中存储的端点、端口等信息：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 [vagrant@master01 rook-ceph]$ kubectl get cm -o yaml ceph-bucket -n obc-demo apiVersion: v1 data: BUCKET_HOST: rook-ceph-rgw-my-store.rook-ceph.svc BUCKET_NAME: ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9 BUCKET_PORT: "80" BUCKET_REGION: "" BUCKET_SUBREGION: "" kind: ConfigMap metadata: creationTimestamp: "2024-03-04T01:48:19Z" finalizers: - objectbucket.io/finalizer labels: bucket-provisioner: rook-ceph.ceph.rook.io-bucket name: ceph-bucket namespace: obc-demo ownerReferences: - apiVersion: objectbucket.io/v1alpha1 blockOwnerDeletion: true controller: true kind: ObjectBucketClaim name: ceph-bucket uid: c47194f4-8ab1-48c1-b436-2ac6c8313eb4 resourceVersion: "181925" uid: e6c1cb11-de23-4381-9b7d-0ffbc64d0518
所以应用需要访问对象存储，只需将 OBC 对应的 secret 和 configmap 注入到 Pod 中即可。
使用 s5cmd 访问对象存储 toolbox Pod 不提供 s5cmd，可以部署 toolbox-operator-image ，里面包含了该客户端工具。deploy/examples/toolbox-operator-image.yaml 中：
1 2 3 4 5 [vagrant@master01 rook]$ kubectl apply -f deploy/examples/toolbox-operator-image.yaml deployment.apps/rook-ceph-tools-operator-image created [vagrant@master01 rook]$ kubectl get pod -n rook-ceph | grep tools-operator rook-ceph-tools-operator-image-5df88d9f87-tb7zt 1/1 Running 0 38s
配置 s5cmd 在 toolbox 中配置对象存储，需要注意的是 Secret 中的信息是 base64 加密的，如果直接通过 get -o yaml 方式，需要解密：
1 2 3 4 5 6 7 [vagrant@master01 rook]$ kubectl exec -it -n rook-ceph rook-ceph-tools-operator-image-5df88d9f87-tb7zt -- /bin/bash [rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt /]$ mkdir ~/.aws [rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt /]$ cat > ~/.aws/credentials <<EOF > [default] > aws_access_key_id = QDURTCLD1SF1LMLW1V5K > aws_secret_access_key = 2x04kP828eK6my7v1gIBQzP9oFZttRTnf9gnk0Qq > EOF
验证凭据是否正确：
1 2 [rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ s5cmd --endpoint-url http://rook-ceph-rgw-my-store.rook-ceph.svc ls 2024/03/04 01:48:19 s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9
PUT 和 GET 对象 上传一个新对象到对象存储中：
1 2 3 4 5 6 7 [rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt /]$ echo "Hello Rook!" > /tmp/rookObj [rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ s5cmd --endpoint-url http://rook-ceph-rgw-my-store.rook-ceph.svc cp /tmp/rookObj s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9 cp /tmp/rookObj s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9/rookObj[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ s5cmd --endpoint-url http://rook-ceph-rgw-my-store.rook-ceph.svc ls s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9 2024/03/04 02:15:36 12 rookObj
从对象存储中下载对象：
1 2 3 4 [rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ s5cmd --endpoint-url http://rook-ceph-rgw-my-store.rook-ceph.svc cp s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9/rookObj /tmp/download_obj cp s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9/rookObj /tmp/download_obj[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ diff /tmp/rookObj /tmp/download_obj
评论· · · · · ·