「PVE」- 存储(Storage)

  CREATED BY JENKINSBOT

磁盘扩容

Resize disks – Proxmox VE

qm resize 100 virtio0 +5G                                                       # 将磁盘增加 5G
qm resize 100 virtio0  5G                                                       # 将磁盘设为 5G

参考 Resize disks – Proxmox VE 文档,以获取后续的 磁盘扩容、文件系统扩容 相关内容。

NFS

unable to activate storage ‘iso-image’ – directory ‘xxx’ does not exist or is unreachable (500)

[SOLVED] – Storage status unknown | Proxmox Support Forum
[SOLVED] – Storage status unknown | Proxmox Support Forum

问题描述:突然发现 NFS 无法正常访问,提示如上错误;

原因分析:我们曾修改 PVE 管理接口的地址,由于 NFS 限制,导致我们无法正常访问;

解决方案:调整 NFS 访问控制,允许新地址进行访问;

超融合(Ceph)

Deploy Hyper-Converged Ceph Cluster – Proxmox VE

通过命令行安装 Ceph 存储:

# pveceph install

创建 OSD 实例

需要选择节点,然后再添加磁盘;

删簇 OSD 实例

Proxmox Ceph remove OSD – How to do it via Proxmox VE GUI and CLI?

通过图形化界面删除:
1)Firstly, we select the Proxmox VE node in the tree.
2)Next, we go to Ceph >> OSD panel. Then we select the OSD to remove. And click the OUT button.
3)When the status is OUT, we click the STOP button. This changes the status from up to down.
4)Finally, we select the More drop-down and click Destroy.

通过命令行删除:
1)ceph osd out {osd-num}
2)查看 OSD 数据迁移状态:ceph -w
3)systemctl stop ceph-osd@{osd-num}
4)ceph osd crush remove {name}
5)ceph auth del osd.{osd-num}
6)ceph osd rm {osd-num}

清理为删除的 Monitor 实例

[SOLVED] – CEPH how to delete dead monitor? | Proxmox Support Forum

如果为清理 Ceph 而直接删除节点,则会导致 Monitor 状态异常。

删除未使用的 Monitor 实例:

# ceph -s
...

# ceph mon remove xxxxxxx

问题:虽然 Monitor 已删,但是仍旧在 GUI 中显示;

[SOLVED] – Ghost monitor in CEPH cluster | Page 2 | Proxmox Support Forum
linux – How to remove systemd services – Super User
Re-adding Ceph Node | Proxmox Support Forum
[SOLVED] – CEPH how to delete dead monitor? | Proxmox Support Forum
Reinstall/remove dead monitor | Proxmox Support Forum

systemctl disable ceph-mon@<name-of-mon>.service
rm -rf /etc/systemd/system/ceph-mon.target.wants/ceph-mon@<name-of-mon>.service

rm -rf /var/lib/ceph/mon/<name of monitor>

vim /etc/pve/ceph.conf
...(1)删除对应的 Monitor 条目;
...(2)删除 [global].mon_host 中对应地址;

常见问题处理

所有 LV 进入 INACTIVE 状态

local-LVM not available after Kernel update on PVE 7 | Page 2 | Proxmox Support Forum
2023213 – Autoactivation of thin pool during boot fails

问题描述:
当物理节点重启后,该节点虚拟机无法正常启动,虚拟机实例的磁盘进入 INACTIVE 状态;

原因分析:
thin_check 执行时间过长,导致 pvscan 被结束。

解决方案:

# 问题修复
lvchange -an pve/data_tmeta
lvchange -an pve/data_tdata
vgchange -ay pve

# Workaround
vim /etc/lvm/lvm.conf
... thin_check_options = [ "-q", "--skip-mappings" ]
update-initramfs -u

创建 OSD 失败:… unable to find a keyring … entity osd.0 exists but key does not match …

[SOLVED] – PVE7 unable to create OSD | Proxmox Support Forum
entity osd.5 exists but key does not match | Proxmox Support Forum

在添加 OSD 设备时,提示如下错误:

create OSD on /dev/sdi (bluestore)
wiping block device /dev/sdi
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.464955 s, 451 MB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 435258e3-31c8-4fc7-958e-3d37f881e388
 stderr: 2022-10-13T01:37:26.795+0800 7f9276af5700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
 stderr: 2022-10-13T01:37:26.795+0800 7f9276af5700 -1 AuthRegistry(0x7f927005b868) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
 stderr: Error EEXIST: entity osd.0 exists but key does not match
-->  RuntimeError: Unable to create a new OSD id
TASK ERROR: command 'ceph-volume lvm create --cluster-fsid fc27b60a-41ba-4540-987e-0cd0f7d1b1f4 --data /dev/sdi' failed: exit code 1

按照 [SOLVED] – PVE7 unable to create OSD | Proxmox Support Forum 提示,进行修复:

cp /etc/pve/priv/ceph.client.bootstrap-osd.keyring /etc/pve/priv/ceph.client.bootstrap-osd.keyring.backup # 该文件应该是不存在的
ceph auth get client.bootstrap-osd > /etc/pve/priv/ceph.client.bootstrap-osd.keyring

但是,再次添加 OSD 设备,提示如下错误:

create OSD on /dev/sdi (bluestore)
wiping block device /dev/sdi
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.47554 s, 441 MB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a2f8be83-dfc5-4843-a984-a511f815f96b
 stderr: Error EEXIST: entity osd.0 exists but key does not match
-->  RuntimeError: Unable to create a new OSD id
TASK ERROR: command 'ceph-volume lvm create --cluster-fsid fc27b60a-41ba-4540-987e-0cd0f7d1b1f4 --data /dev/sdi' failed: exit code 1

按照 entity osd.5 exists but key does not match | Proxmox Support Forum 提示,我们发现是操作不规范:旧的 OSD 没有正确删除;

解决方案:ceph auth del osd.0