「Ceph」- 配置对象存储(Object Storage,S3,Swift)

  CREATED BY JENKINSBOT

解决方案

在实际的生产环境中:
1)建立部署独立的 RGW 服务,而不要与 MON 共用主机(除非负载较小);
2)建议部署负载均衡,在负载均衡后端挂接多个 RGW 实例;

环境概述

Ceph Cluster (with Three Nodes)
     ||
  Ceph RGW
     ||
 (S3, Swift)
     ||
  Ceph User

Ceph Cluster: ceph-node-01 172.31.252.201; ceph-node-02 172.31.252.202; ceph-node-03 172.31.252.203;
Ceph RGW: ceph-rgw-01 172.31.252.209;
Ceph Client: ubuntu-developing 172.31.252.100;

我们将在 ceph-rgw-01 中部署 RGW 服务,并通过 ubuntu-developing 访问对象存储;

第一步、配置网关服务

同步配置文件

on ceph-node-01

# cephadm install ceph-common
# cephadm install radosgw

# scp /etc/ceph/ceph.conf 192.168.200.9:/etc/ceph/

添加网关凭证

目标:实现 RGW 访问 Ceph Cluster 的配置;

on ceph-node-01

// 创建 Keyring 信息

ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
chmod +r /etc/ceph/ceph.client.radosgw.keyring 

ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key
ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring 

// 添加 Keyring 到集群

ceph auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring 

// 分配 Keyring 给节点

scp /etc/ceph/ceph.client.radosgw.keyring 172.31.252.209:/etc/ceph/ceph.client.radosgw.keyring

on ceph-rgw-01

// ---------------------------------------------------------------------------- // 配置 RGW 实例

# vim /etc/ceph/ceph.conf
[client.radosgw.gateway]
host = ceph-rgw-01
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
rgw dns name = ceph-rgw-01.dc-laptop.d3rm.org
rgw print continue = false

关于 rgw dns name 参数:所指定的域名 及 其子域名 要能够解析到当前节点,否则 S3 将无法访问对象存储;

启动网关服务

// ---------------------------------------------------------------------------- // 启动 RGW 服务

# radosgw.gateway 为 client 的后半部分(参考 systemctl cat ceph-radosgw@.service 输出);
systemctl start ceph-radosgw@radosgw.gateway.service
netstat -npl | grep 7480

第二步、创建访问用户

目标:实现 User 访问 RGW 的配置;

针对对象存储的访问:
1)若通过 S3 API 访问,则创建 S3 用户接口;
2)若通过 Swift 访问,则需要先创建 S3 接口的网关用户,然后在创建 Swift 接口的子用户;

鉴于 Ceph Swift 并非我们关注的重点,亦非我们当前技术栈之内,所相关内容较少提及;

on ceph-rgw-01

创建 S3 用户

# radosgw-admin -k /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway \
    user create --uid=mona --display-name="Monika"
{
    "user_id": "mona",
    "display_name": "Monika",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "mona",
            "access_key": "QP0ZJMOUW8H39JUYM9QY",
            "secret_key": "wrteSg9xIeNp7ukewOuMxaMay9zWt02ZNskop7mj"
        }
    ],
    ...
}

创建 Swift 子用户

# radosgw-admin -k /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway \
    subuser create --uid=mona --subuser=mona:swift --access=full
{
    ...
    "subusers": [
        {
            "id": "mona:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "mona",
            "access_key": "QP0ZJMOUW8H39JUYM9QY",
            "secret_key": "wrteSg9xIeNp7ukewOuMxaMay9zWt02ZNskop7mj"
        }
    ],
    "swift_keys": [
        {
            "user": "mona:swift",
            "secret_key": "EA2FdtN2kuPVb0hDSS7mqlrYyw3tcnFBvhBAn5Ld"
        }
    ],
    ...
}

// 为子用户创建密钥

# radosgw-admin -k /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway \
    key create --subuser=mona:swift --key-type=swift --gen-secret
{
    ...
    "subusers": [
        {
            "id": "mona:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "mona",
            "access_key": "QP0ZJMOUW8H39JUYM9QY",
            "secret_key": "wrteSg9xIeNp7ukewOuMxaMay9zWt02ZNskop7mj"
        }
    ],
    "swift_keys": [
        {
            "user": "mona:swift",
            "secret_key": "ZyyCp2WHc3LjbNBnOEV3OzuARIQqbZQEBxmFBItl"
        }
    ],
    ...
}

第三步、访问对象存储

通过 S3 访问

# apt-get install s3cmd

# s3cmd --configure
...
New settings:
  Access Key: QP0ZJMOUW8H39JUYM9QY
  Secret Key: wrteSg9xIeNp7ukewOuMxaMay9zWt02ZNskop7mj
  Default Region: US
  S3 Endpoint: ceph-rgw-01.dc-laptop.d3rm.org:7480
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.ceph-rgw-01.dc-laptop.d3rm.org:7480
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0
...

# s3cmd mb s3://testing
Bucket 's3://testing/' created

# s3cmd ls
ERROR: S3 error: 403 (SignatureDoesNotMatch)

# s3cmd put /etc/hosts s3://testing
upload: '/etc/hosts' -> 's3://testing/hosts'  [1 of 1]
 232 of 232   100% in    2s   104.72 B/s  done

# s3cmd ls s3://testing
2022-11-14 09:41       232   s3://testing/hosts

# s3cmd get s3://testing/hosts /tmp/hosts.s3
download: 's3://testing/hosts' -> '/tmp/hosts.s3'  [1 of 1]
 232 of 232   100% in    0s    66.99 kB/s  done

通过 Swift 访问

鉴于 Ceph Swift 并非我们关注的重点,亦非我们当前技术栈之内,所相关内容较少提及;

# apt-get install python3-swiftclient

# radosgw-admin -k /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway user info --uid="mona"
{
    ...
    "swift_keys": [
        {
            "user": "mona:swift",
            "secret_key": "ZyyCp2WHc3LjbNBnOEV3OzuARIQqbZQEBxmFBItl"
        }
    ],
    ...
}

# swift -A http://172.31.252.209:7480/auth/1.0 -U mona:swift -K ZyyCp2W... list
testing.ceph-rgw-01.dc-laptop.d3rm.org

# swift -A http://172.31.252.209:7480/auth/1.0 -U mona:swift -K ZyyCp2W... post second-bucket

# swift -A http://172.31.252.209:7480/auth/1.0 -U mona:swift -K ZyyCp2W... list
second-bucket
testing.ceph-rgw-01.dc-laptop.d3rm.org