How to: Fix Proxmox VE/ZFS Pool extremely slow write performance issue

The Issue

If we just create the ZFS pool from Proxmox gui, then start to use it. (Especially for HDDs)

e.g. We write large datasets continuously.

Sooner or later (Depend on the ZFS pool usage), we will find out that the writes is around 1-10MB/s, which is extremely slow.

(Note: If we test the newly created pool with and without ZIL/SLOG, there probably won’t be much difference or even slower with ZIL/SLOG deice attached, after we have data filled in the ZFS pool, the pool with dedicate ZIL/SLOG device will perform better than the pool without dedicate ZIL/SLOG device)

The Fix

This can happen due to ZFS Intent Log (ZIL)/Separate ZFS Intent Log (SLOG) is getting written to the same ZFS data pool which all our data are stored, which eventually caused “double write” issue.

To fix this issue is easy, best way to fix it properly is to grab a SSD, worst case, if we do not have one temporary, we can even grab a 5400RPM or 7200RPM HDD use it via HBA/SATA/SCSI or even USB 3.0 (USB is not suitable for long term for this purpose, but can work fine as an temporary solution/fix).

Once attached the disk to the Proxmox host, note down the device name e.g. /dev/sde, /dev/sdf etc.

Login to terminal from Proxmox host or via SSH or via Shell from web gui.

Use following command to use an dedicated HDD/SSD for ZIL/SLOG purpose

# For single HDD/SSD
zpool add -f [pool name] log [device name]
# e.g.
zpool add -f rpool log /dev/sdd
 
# For mirrored ZIL/SLOG
zpool add -f [pool name] log mirror [device 1 name] [device 2 name]
# e.g.
zpool add -f rpool log mirror /dev/sdd /dev/sde

Now if we have a look at write performance, it will be increased

Note: Best device for ZIL/SLOG is to have an datacenter grade SSD via HBA, so that we can get best performance

To check the pool status see if the ZIL/SLOG device is added use following commands

zpool status

Bonus

Read/Write buffer/cache for ZFS pool

With ZFS pool

  • SLOG is used to cache synchronous ZIL data (Write) before flushing to disk (Write performance related)
  • Adaptive Replacement Cache (ARC) and Second level adaptive replacement cache (L2ARC) are used to cache reads (Read performance related)

How big should the ZIL/SLOG device/HDD/SSD be?

Usually the ZIL is default to flush every 5 seconds or when it reaches capacity, which means a SLOG that holds 5-10 seconds’ worth of the pool maximum throughput will be find, unless we are doing something extreme/special.

How to Remove/Delete/Replace ZIL/SLOG device/HDD/SSD?

# To remove ZIL/SLOG device
zpool remove [pool name] [device name]
# e.g.
zpool remove rpool /dev/sdd
or
zpool remove rpool sdd
or
# By using real device name
zpool remove rpool ata-xxxx_xxxx_Xxxx_x.....

To replace, simply remove the current HDDs/SSDs then add new disks again

How to find out real device name from Proxmox?

Refer to this guide: How to: Find drive name (real name) for /dev/sdb /dev/sdc from Proxmox (PVE)

How to add read cache disks?

It is very similar to adding ZIL/SLOG drives

zpool add -f [pool name] cache [device name]
# e.g.
zpool add -f rpool cache /dev/sde
 
# Remove cache disk
zpool remove [pool name] [device name]
zpool remove rpool /dev/sde
zpool remove rpool /sde

If the above remove command does not work, try remove the ZIL/SLOG first


How to: Add/Remove/Delete independent/dedicate Intent Log Device (ZIL)/Separate Intent Log (SLOG) drive for ZFS (Proxmox (PVE))

1 Login to terminal via direct login, web gui or SSH to Proxmox

Adding drive as ZIL/SLOG device

zpool add -f poolName log deviceName
 
# e.g.
zpool add -f rpool log /dev/sdb

(To add mirrored ZIL/SLOG drives)

zpool add -f rpool log mirror /dev/sdb /dev/sdc

-f: Without -f flag, we will get a warning about missing EFI label

Use following command to check ZIL/SLOG status

zpool status

Note: If the ZIL/SLOG device failed, we will lose seconds worth of writes but our file system will continue to function without data corruption.

Remove/Delete ZIL/SLOG drive

zpool remove poolName deviceName
 
# e.g.
zpool remove rpool /dev/sdb
 
#By using drive's real name rather than /dev/sdb
zpool remove rpool ata-HGST_HTSxxxxxxx_RCFxxxxxxxxx-part9

For finding real drive names: How to: Find drive name (real name) for /dev/sdb /dev/sdc from Proxmox (PVE)