Skip to content
Proxmoxes
ZFS Setup And Management

ZFS Setup And Management

ZFS Pool Setup and Management

This guide covers the main commands required to create, inspect, maintain, expand, troubleshoot, and destroy ZFS pools on Linux/Proxmox.

Warning: Commands such as zpool create, zpool destroy, zfs destroy, and disk wiping commands can permanently erase data. Always verify device names first.

1. Identify Disks

List disks:

1

lsblk

Show disk model and serial numbers:

1

lsblk -o NAME,SIZE,MODEL,SERIAL

List persistent disk IDs:

1

ls -l /dev/disk/by-id/

For ZFS pools, prefer persistent disk paths such as:

1

/dev/disk/by-id/wwn-0x5000c500xxxxxxxx

rather than:

1

/dev/sda

Creating a ZFS Pool

RAIDZ2 Example

Example using 8 disks:

1
2
3
4
5
6
7
8
9

zpool create storage raidz2 \
  /dev/disk/by-id/wwn-DISK1 \
  /dev/disk/by-id/wwn-DISK2 \
  /dev/disk/by-id/wwn-DISK3 \
  /dev/disk/by-id/wwn-DISK4 \
  /dev/disk/by-id/wwn-DISK5 \
  /dev/disk/by-id/wwn-DISK6 \
  /dev/disk/by-id/wwn-DISK7 \
  /dev/disk/by-id/wwn-DISK8

Specify a mount point:

1
2
3
4
5

zpool create -m /srv/storage storage raidz2 \
  /dev/disk/by-id/wwn-DISK1 \
  /dev/disk/by-id/wwn-DISK2 \
  /dev/disk/by-id/wwn-DISK3 \
  /dev/disk/by-id/wwn-DISK4

Recommended Pool Properties

Enable automatic trimming for SSD pools:

1

zpool set autotrim=on storage

Check:

1

zpool get autotrim storage

Enable automatic expansion:

1

zpool set autoexpand=on storage

Check:

1

zpool get autoexpand storage

Creating ZFS Datasets

Create a dataset:

1

zfs create storage/media

Create additional datasets:

1
2
3

zfs create storage/media
zfs create storage/backups
zfs create storage/vmdata

List datasets:

1

zfs list

Dataset Mount Points

Check mount point:

1

zfs get mountpoint storage/media

Set mount point:

1

zfs set mountpoint=/srv/media storage/media

Disable automatic mounting:

1

zfs set canmount=off storage/media

Re-enable:

1

zfs set canmount=on storage/media

Mount dataset:

1

zfs mount storage/media

Mount all datasets:

1

zfs mount -a

Unmount:

1

zfs unmount storage/media

Compression

Enable compression:

1

zfs set compression=lz4 storage

Or on a specific dataset:

1

zfs set compression=lz4 storage/media

Check compression:

1

zfs get compression storage/media

Check compression ratio:

1

zfs get compressratio storage/media

Record Size

Check:

1

zfs get recordsize storage/media

For large media files:

1

zfs set recordsize=1M storage/media

For general use:

1

zfs set recordsize=128K storage/data

Record size affects newly written data only.

Atime

Disable access-time updates:

1

zfs set atime=off storage

Check:

1

zfs get atime storage

This can reduce unnecessary writes.

Pool Status

Basic status:

1

zpool status

Specific pool:

1

zpool status storage

Verbose status:

1

zpool status -v storage

Show persistent/full disk paths:

1

zpool status -P storage

Pool Capacity

1

zpool list

More detailed dataset usage:

1

zfs list

Show all properties:

1

zfs get all storage

ZFS I/O Monitoring

Pool I/O:

1

zpool iostat

Detailed per-disk I/O:

1

zpool iostat -v

Refresh every second:

1

zpool iostat -v 1

Refresh every 5 seconds:

1

zpool iostat -v 5

Show latency statistics where supported:

1

zpool iostat -v -l 1

Scrubbing

Start a scrub:

1

zpool scrub storage

Check scrub progress:

1

zpool status storage

Stop a running scrub:

1

zpool scrub -s storage

A scrub checks all stored data against ZFS checksums and attempts to repair corrupted data using redundancy.

Disk Errors

Check for errors:

1

zpool status -v storage

Clear recorded errors after fixing the cause:

1

zpool clear storage

Clear errors for a specific disk:

1

zpool clear storage /dev/disk/by-id/wwn-DISK

Offline / Online a Disk

Take a disk offline:

1

zpool offline storage /dev/disk/by-id/wwn-DISK

Bring it back online:

1

zpool online storage /dev/disk/by-id/wwn-DISK

Replace a Failed Disk

Find the failed disk:

1

zpool status storage

Replace it:

1
2
3

zpool replace storage \
  /dev/disk/by-id/wwn-OLD-DISK \
  /dev/disk/by-id/wwn-NEW-DISK

Monitor resilvering:

1

zpool status storage

Or:

1

watch zpool status storage

Detach / Remove Devices

For mirrored pools, detach a disk:

1

zpool detach storage /dev/disk/by-id/wwn-DISK

detach is generally for mirrors. It cannot simply remove a member disk from RAIDZ.

Add Another VDEV

Example adding another RAIDZ2 vdev:

1
2
3
4
5

zpool add storage raidz2 \
  /dev/disk/by-id/wwn-DISK9 \
  /dev/disk/by-id/wwn-DISK10 \
  /dev/disk/by-id/wwn-DISK11 \
  /dev/disk/by-id/wwn-DISK12

Adding a vdev permanently changes the pool layout. Verify the command carefully before running it.

Expanding After Replacing Disks

After replacing disks with larger disks:

1

zpool online -e storage /dev/disk/by-id/wwn-DISK

Or enable automatic expansion:

1

zpool set autoexpand=on storage

Snapshots

Create snapshot:

1

zfs snapshot storage/media@backup

List snapshots:

1

zfs list -t snapshot

Create recursive snapshot:

1

zfs snapshot -r storage@backup

Rollback:

1

zfs rollback storage/media@backup

Delete snapshot:

1

zfs destroy storage/media@backup

Snapshot Clones

Clone a snapshot:

1

zfs clone storage/media@backup storage/media-test

Remove clone:

1

zfs destroy storage/media-test

Sending and Receiving ZFS Data

Create a snapshot:

1

zfs snapshot storage/media@backup

Send to another pool:

1

zfs send storage/media@backup | zfs receive backup/media

Send over SSH:

1

zfs send storage/media@backup | ssh root@server zfs receive backup/media

Incremental Replication

Create two snapshots:

1
2

zfs snapshot storage/media@snap1
zfs snapshot storage/media@snap2

Send only the changes:

1
2

zfs send -i storage/media@snap1 storage/media@snap2 | \
zfs receive backup/media

Export a Pool

Useful before moving disks to another machine:

1

zpool export storage

Import a Pool

List available pools:

1

zpool import

Import:

1

zpool import storage

Force import if required:

1

zpool import -f storage

Import using persistent disk IDs:

1

zpool import -d /dev/disk/by-id storage

Rename a Pool

Export it:

1

zpool export storage

Import using the new name:

1

zpool import storage new-storage

Destroy a Dataset

1

zfs destroy storage/media

Destroy recursively:

1

zfs destroy -r storage/media

This permanently deletes the dataset and its data.

Destroy a Pool

1

zpool destroy storage

Warning: This permanently destroys the ZFS pool.

Disk Information

Check SMART information:

1

smartctl -a /dev/sdX

Using a persistent ID:

1

smartctl -a /dev/disk/by-id/wwn-DISK

Short SMART test:

1

smartctl -t short /dev/sdX

Long SMART test:

1

smartctl -t long /dev/sdX

Useful ZFS ARC Commands

View ARC statistics:

1

arc_summary

Live ARC statistics:

1

arcstat 1

Check current ARC size:

1

grep -E '^(size|c_max|c_min) ' /proc/spl/kstat/zfs/arcstats

Check ZFS Version

1

zfs version

Pool History

View commands that have modified the pool:

1

zpool history storage

Verbose history:

1

zpool history -il storage

This is particularly useful when troubleshooting configuration changes.

Common Health Check

A useful quick check:

1
2
3

zpool status
zpool list
zfs list

For detailed live I/O:

1

zpool iostat -v 1

For recent pool operations:

1

zpool history storage

Typical Recommended Dataset Setup

Example:

1
2
3
4
5
6
7
8

zfs create storage/media
zfs create storage/backups
zfs create storage/vmdata

zfs set compression=lz4 storage
zfs set atime=off storage

zfs set recordsize=1M storage/media

Result:

1
2
3
4

storage
├── media
├── backups
└── vmdata

Recommended Routine Maintenance

Check pool health:

1

zpool status

Check capacity:

1
2

zpool list
zfs list

Monitor disk activity:

1

zpool iostat -v 1

Run a scrub:

1

zpool scrub storage

Monitor scrub:

1

zpool status storage

Check SMART health periodically:

1

smartctl -a /dev/disk/by-id/wwn-DISK

For important storage, schedule regular scrubs and SMART tests rather than relying on manual checks alone.

QEMU Guest Agent Install