ZPOOL 存储池更换硬盘
现状
我的 NAS 机箱只有 4 个盘位，我的存储池结构如下：
root@nas:~ # zpool status -v
pool: house
state: ONLINE
scan: scrub canceled on Tue Dec 12 21:03:48 2023
config:
NAME STATE READ WRITE CKSUM
house ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
diskid/DISK-WFN4LS7E ONLINE 0 0 0
diskid/DISK-WFN4MVMP ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
diskid/DISK-S1Z112EF ONLINE 0 0 0
diskid/DISK-WJG09SV8 ONLINE 0 0 0
errors: No known data errors其中 diskid/DISK-S1Z112EF 要出问题了，需要更换这块盘。现有这 4 块盘都是 4T 硬盘，我采购了两块 8T 硬盘，打算把 mirror-1 的盘换掉，顺便扩一下空间。
目标
最终的状态是这个存储池由两个镜像组成，每个镜像两块盘：
NAME STATE READ WRITE CKSUM
house ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
diskid/DISK-WFN4LS7E ONLINE 0 0 0
diskid/DISK-WFN4MVMP ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
diskid/新1
diskid/新2理想计划
更换硬盘应该很简单，拔下 S1Z112EF 这块盘
然后插入新盘
用 zpool replace 就行
遇到问题
但在执行的时候，报了个错：
root@nas:~ # zpool replace -f house 17180553841587689187 /dev/diskid/DISK-VGJRPGGK
cannot replace 17180553841587689187 with /dev/diskid/DISK-VGJRPGGK: already in replacing/spare config; wait for completion or use 'zpool detach'没查出原因，然后我把 S1Z112EF 这个盘 detach 了，现在的硬盘状态如下：
root@nas:~ # zpool detach house 17180553841587689187
root@nas:~ # zpool status -v
pool: house
state: ONLINE
scan: scrub canceled on Tue Dec 12 21:03:48 2023
config:
NAME STATE READ WRITE CKSUM
house ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
diskid/DISK-WFN4LS7E ONLINE 0 0 0
diskid/DISK-WFN4MVMP ONLINE 0 0 0
diskid/DISK-WJG09SV8 ONLINE 0 0 0
errors: No known data errors我想着把新盘加入，和 WJG09SV8 这个盘做个镜像，又出错了：
root@nas:~ # zpool attach house diskid/DISK-WJG09SV8 /dev/diskid/DISK-VGJRPGGK
cannot attach /dev/diskid/DISK-VGJRPGGK to diskid/DISK-WJG09SV8: can only attach to mirrors and top-level disks又是一痛查询，原来是因为 ashift 属性值不对：
root@nas:~ # zdb | grep ashift
ashift: 12
ashift: 9上面的 mirror-0 的 ashift 值为 12，下面的 WJG09SV8（原来的 mirror-1）值是 9，我的新盘是 12，这个 ashift 值的意思是扇区大小，值 9 表示 512B，值 12 表示 4K，值 13 表示 8K。
新盘和旧盘值不同，不同就不能加，如果硬要加呢，在 zpool attach 后要用-o 来指定 ashift 值，要用小的。为了以后着想，我还是想把 9 的去掉，但 ZFS 不能直接改。
新计划
用新盘做一个新池
数据复制到新池
新池做成镜像
实操
我计划用新盘做一个新的池，把旧存储池的数据复制过去。
root@nas:~ # zpool create nas /dev/diskid/DISK-VGJRPGGK
root@nas:~ # zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
house 7.25T 5.81T 1.44T - - 29% 80% 1.00x ONLINE -
nas 7.27T 468K 7.27T - - 0% 0% 1.00x ONLINE -存储池建好了，我的 zfs 如下：
root@nas:~ # zfs list
NAME USED AVAIL REFER MOUNTPOINT
house 5.81T 1.31T 71K /house
house/bt 4.35T 1.31T 4.35T /house/bt
house/data 858G 1.31T 858G /house/data
house/docs 640G 1.31T 640G /house/docs
nas 384K 7.14T 96K /nas使用 zfs send 和 zfs receive 来复制，先要创建快照，再复制：
root@nas:~ # zfs snapshot house/docs@20231216
root@nas:~ # zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
house/docs@20231216 0B - 640G -
root@nas:~ # zfs send -R house/docs@20231216 | zfs receive nas/docs数据复制完成后，新的存储池（nas）下就会出现新的 ZFS 分区。
root@nas:~ # df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ada3p2 899G 139G 688G 17% /
devfs 1.0K 1.0K 0B 100% /dev
house 1.3T 71K 1.3T 0% /house
house/docs 1.9T 640G 1.3T 32% /house/docs
house/data 2.2T 858G 1.3T 39% /house/data
house/bt 5.7T 4.3T 1.3T 77% /house/bt
nas 6.5T 96K 6.5T 0% /nas
nas/docs 7.1T 640G 6.5T 9% /nas/docs依次复制其他的 ZFS 分区。
经过漫长的等待，数据已全部复制到新的池里了
root@nas:~ # zfs list
NAME USED AVAIL REFER MOUNTPOINT
house 5.81T 1.31T 71K /house
house/bt 4.35T 1.31T 4.35T /house/bt
house/data 858G 1.31T 858G /house/data
house/docs 640G 1.31T 640G /house/docs
nas 5.81T 1.33T 96K /nas
nas/bt 4.35T 1.33T 4.35T /nas/bt
nas/data 858G 1.33T 858G /nas/data
nas/docs 640G 1.33T 640G /nas/docs下一步把旧池导出
root@nas:~ # zpool export house关机，拔下旧盘DISK-WJG09SV8，插上新盘，再和新池里的盘组成镜像。
root@nas:~ # zpool attach nas diskid/DISK-VGJRPGGK /dev/diskid/DISK-VGK62G1G等着同步完成
root@nas:~ # zpool status -v
pool: nas
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Dec 18 07:28:08 2023
1.91T scanned at 166M/s, 1.53T issued at 133M/s, 5.81T total
1.53T resilvered, 26.26% done, 09:24:17 to go
config:
NAME STATE READ WRITE CKSUM
nas ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
diskid/DISK-VGJRPGGK ONLINE 0 0 0
diskid/DISK-VGK62G1G ONLINE 0 0 0 (resilvering)
errors: No known data errors
root@nas:~ # zpool status -v
pool: nas
state: ONLINE
scan: resilvered 5.82T in 11:32:22 with 0 errors on Mon Dec 18 19:00:30 2023
config:
NAME STATE READ WRITE CKSUM
nas ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
diskid/DISK-VGJRPGGK ONLINE 0 0 0
diskid/DISK-VGK62G1G ONLINE 0 0 0
errors: No known data errors
同步完成后加入原池中两块好的盘：
root@nas:~ # zpool add -f nas mirror /dev/diskid/DISK-WFN4LS7E /dev/diskid/DISK-WFN4MVMP
root@nas:~ # zpool status -v
pool: nas
state: ONLINE
scan: resilvered 5.82T in 11:32:22 with 0 errors on Mon Dec 18 19:00:30 2023
config:
NAME STATE READ WRITE CKSUM
nas ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
diskid/DISK-VGJRPGGK ONLINE 0 0 0
diskid/DISK-VGK62G1G ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
diskid/DISK-WFN4LS7E ONLINE 0 0 0
diskid/DISK-WFN4MVMP ONLINE 0 0 0
errors: No known data errors改存储池名：
root@nas:~ # zpool export nas
root@nas:~ # zpool import nas house
root@nas:~ # zpool status -v
pool: house
state: ONLINE
scan: resilvered 5.82T in 11:32:22 with 0 errors on Mon Dec 18 19:00:30 2023
config:
NAME STATE READ WRITE CKSUM
house ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
diskid/DISK-VGJRPGGK ONLINE 0 0 0
diskid/DISK-VGK62G1G ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
diskid/DISK-WFN4LS7E ONLINE 0 0 0
diskid/DISK-WFN4MVMP ONLINE 0 0 0
errors: No known data errors