How to: Backup and Restore Proxmox VE host

Warning: This guide is to backup Proxmox VE host, not virtual machines, Proxmox VE has built-in VM backup mechanism which can be configured easily via its web gui.

There are many ways to backup Proxmox VE host, e.g. using xfsdump, using different kinds of third party backup programs like rsync etc.

For this guide we will be using UrBackup

Pre-requirements

  • A working Proxmox VE host
  • A spare server for running as UrBackup server with big enough storage
  • The Proxmox VE host and the UrBackup Server is connected and in the same LAN

1 Install UrBackup Server

The server can be Windows or Linux (Debian/Ubuntu etc.)

1.1 For Windows Server

1.1.1 Download the exe or msi file from here: UrBackup Windows Server [1]

1.1.2 Execute the downloaded binary file, install the UrBackup server

1.1.3 Open any browser from the server, navigate to “localhst:55414” or “127.0.0.1:55414”

Note 1: The web interface is available at port 55414. To restrict the access you have to create an admin account in Settings->Users. Without this account everyone can access all backups using the web interface.

Note 2: This web interface is accessible from other device within the same LAN

1.2 For Linux (Ubuntu) Server

1.2.1 We can simply use following commands to install it

sudo add-apt-repository ppa:uroni/urbackup
sudo apt update
sudo apt install urbackup-server

or

wget https://hndl.urbackup.org/Server/2.4.12/urbackup-server_2.4.12_amd64.deb
sudo dpkg -i urbackup-server_2.4.12_amd64.deb
sudo apt install -f

Note 1: It has some dependencies which you can automatically resolve by running apt-get -f install. If it does not work you probably chose the wrong packet from stable/testing/unstable. [2]

Note 2: The web interface is available at port 55414. To restrict the access you have to create an admin account in Settings->Users. Without this account everyone can access all backups using the web interface. [2]

Note 3: This web interface is accessible from other device within the same LAN

2 Install UrBackup Client on Proxmox VE host

2.1 Login to Proxmox VE terminal directly or via SSH or via web gui -> Shell

2.2 Execute following command to install UrBackup client

TF=$(mktemp) && wget "https://hndl.urbackup.org/Client/2.4.10/UrBackup%20Client%20Linux%202.4.10.sh" -O $TF && sh $TF; rm -f $TF

Note: Unless you are sure you want to use snapshot, dattobd require manual installation, LVM requires free space that equals to the size we are backing up. Choose not to use snapshot here if you are not sure even it says you can use dattobd and LVM.

3 Configure the UrBackup Server (and Proxmox host)

3.1 Before we start to configure UrBackup server, we need to make sure Proxmox VE host’s firewall is disabled, or we will have to open ports for UrBackup server, where are listed in following tables

The Server binds to following default ports:

PortUsageIncoming/OutgoingProtocol
55413FastCGI for web interfaceIncomingTCP
55414HTTP web interfaceIncomingTCP
55415Internet clientsIncomingTCP
35623UDP broadcasts for discoveryOutgoingUDP
[3]

The Client binds to following default ports (all incoming):

PortUsageProtocol
35621Sending files during file backups (file server)TCP
35622UDP broadcasts for discoveryUDP
35623Commands and image backupsTCP
[3]

3.2 Login to UrBackup web interface

3.3 Check if the Proxmox VE host appear under “Status” tab, if not, wait 30 – 60 minutes, if we can see the Proxmox VE is displayed under status tab, proceed to next step

3.4 Click on “Settings”

3.5 Click on “Clients”, then click on the name of Proxmox VE host

3.6 Check “Separate settings for this client”, make necessary changes for “File Backups”, no need to worry about “Image Backups” since it is not supported on Linux. (Do not forget to click on “Save” button)

Important: Make sure for File Backups, we exclude zfs pool if we are backing up the root “/”

e.g. Backing up the root file system “/”

Excluded files (with wildcards): /rpool/*;/mnt/proxmox-hostname/*;/tmp/*
Default directories to backup: /

Note: You probably want to exclude “/mnt/data/*” as well.

If we are not doing any modification to system files, we can just backup following folder

/etc

What’s inside

# Note: replace <proxmox-hostname> with your node name
 
# Virtual machine .conf, LXC .conf, SSL certificates, storage related configuration, system settings etc.
/etc/pve/nodes/<proxmox-hostname>/
  
# Virtual machine .conf files
/etc/pve/nodes/<proxmox-host-name>/qemu-server/
 
# LXC .conf files
/etc/pve/nodes/<proxmox-hostname>/lxc/
 
# Mounting information
/etc/fstab
 
/root
 
/usr/local
 
/var

3.7 Click on “Client”, configure a value for “Soft client quota” if you have very limited storage, probably should modify “File Backup” frequency and other numbers too if storage is very limited.

3.8 If everything is set, we can now start to backup.

(Note: We can use “urbackupclientctl stauts” command from Proxmox VE host to check UrBackup status)

4 Restore

To restore the Proxmox VE host, we just simply perform following steps

4.1 Install the same Proxmox VE version on the new hard drive

4.2 Install UrBackup client again on Proxmox VE host (Make sure the hostname is the same), and the client connectes to the server

Note: If we are restoring to a different hardware, we need to make sure that the hostname is the same and we need the tokens in “/usr/local/var/urbackup/tokens” [4], or we can use “Download client for Linux” button from UrBackup web gui, then the token will be included

4.3 There are two ways to restore file and folders, one is through server web gui, another is via client command with “urbackupclientctl ….”

  • If we are using the client command on client, use “urbackupclientctl –help” to check available options, “urbackupclientctl browse” to browse backups and “urbackupclientctl restore-start” to restore

References

[1] “UrBackup – Download UrBackup for Windows, GNU/Linux or FreeBSD”, Urbackup.org, 2020. [Online]. Available: https://www.urbackup.org/download.html

[2] “UrBackup – Install UrBackup on Debian/Ubuntu”, Urbackup.org, 2020. [Online]. Available: https://www.urbackup.org/debianserverinstall.html

[3] “UrBackup – Server administration manual”, Urbackup.org, 2020. [Online]. Available: https://www.urbackup.org/administration_manual.html#x1-9000010.3

[4] “Restoring files to a different VM / PC”, UrBackup – Discourse, 2016. [Online]. Available: https://forums.urbackup.org/t/restoring-files-to-a-different-vm-pc/2221


How to: Upload ISO files to Proxmox VE (PVE)

So that we can install operating systems 😉

1 Login to Proxmox VE web gui

2 Find the storage with “ISO image” listed in “Content”

Proxmox VE Storage View
Proxmox VE Storage View
Proxmox VE Storage View - Summary - Content - ISO Image
Proxmox VE Storage View – Summary – Content – ISO Image

Note: If the “ISO image” is not listed, that means we can not upload ISO to that storage location (Usually we can create an folder on that storage, make sure “ISO image” is selected during Directory creation, then we can upload ISO files to that storage location)

3 Click on “Content”

4 Click on “Upload” button at the top

5 Make sure “ISO image” is selected for “Content”

6 Now we can upload ISO files to Proxmox VE

7 The rest will be easy, click on “Select File…” button, select the ISO file we want to upload, click on “Upload” button to begin uploading, do not close the page until it’s finished

8 Now we can start to use it to install operating system.


How to: Fix Proxmox VE 6.2.1 Installation error: unknown filesystsms

The Error

error: unknown file systems
error: unknown file systems
error: unknown file systems
error: unknown file systems

Entering rescue mode…

grub rescue>

Usually it appears when we are booting from Proxmox installation USB with a PC/Desktop which is using UEFI and the USB was created via Rufus on Windows.

The Fix

1 Download balenaEtcher (Alternative to Rufus)

2 Use balenaEtcher to load the Proxmox VE iso and make sure only the desired USB flash drive is selected, create the bootable media.

3 Use the Proxmox USB as the same way as we did last time

4 It should load Proxmox installation correctly


How to: Add/Remove/Delete independent/dedicate Intent Log Device (ZIL)/Separate Intent Log (SLOG) drive for ZFS (Proxmox (PVE))

1 Login to terminal via direct login, web gui or SSH to Proxmox

Adding drive as ZIL/SLOG device

zpool add -f poolName log deviceName
 
# e.g.
zpool add -f rpool log /dev/sdb

(To add mirrored ZIL/SLOG drives)

zpool add -f rpool log mirror /dev/sdb /dev/sdc

-f: Without -f flag, we will get a warning about missing EFI label

Use following command to check ZIL/SLOG status

zpool status

Note: If the ZIL/SLOG device failed, we will lose seconds worth of writes but our file system will continue to function without data corruption.

Remove/Delete ZIL/SLOG drive

zpool remove poolName deviceName
 
# e.g.
zpool remove rpool /dev/sdb
 
#By using drive's real name rather than /dev/sdb
zpool remove rpool ata-HGST_HTSxxxxxxx_RCFxxxxxxxxx-part9

For finding real drive names: How to: Find drive name (real name) for /dev/sdb /dev/sdc from Proxmox (PVE)


How to: Delete mounted directory from Proxmox (PVE) (and make it ready for other uses)

1 Login to Proxmox host via web Shell or SSH

2 Navigate to Datacenter -> Storage, delete the directory first

2.1 Navigate to Datacenter -> host name/cluster name -> Directory, note down the Device “/dev/disk/by-uuid/xxxxxxxxxxxxxxx”

3 Execute following command (In this guide, we want to delete mounted folder “/testfolder” on disk “/dev/sdd”)

umount -f /dev/disk/by-uuid/.......
umount -f /dev/sdd

4 Looking for .mount files

cd /etc/systemd/system
ls -ahl

We will see a file with name “mnt-pve-testfolder.mount” or “mnt-pve-test\x2dxfs.mount” the full path is “/etc/systemd/mnt-pve-testfolder.mount”

(If you are not sure which .mount file to delete, use this command to see the content of it “cat mnt-pve-testfolder.mount”, find the .mount file which has the same uuid as the one from Proxmox web gui -> host name/cluster Name -> Directory)

5 Use following command to delete it

rm -f mnt-pve-testfolder.mount

6 Now the mounted directory is removed from “Directory” view from Proxmox

Now we want to make it available for other usages, to achieve this, we have to do following steps

7 Execute following command for reformat the disk

# fdisk
fdisk /dev/sdd
g
w
 
# Format the disk as ext4
mkfs.ext4 /dev/sdd
 
# Format the disk as xfs
mkfs.xfs -f /dev/sdd

Note: If creating directory failed in Proxmox web gui, use “mkfs.ext4” or “mkfs.xfs” command to format the disk to the format that we do not want, then in the creating the directory step in Proxmox, finally reformat to the format that we actually need

8 Now we can go back to the Proxmox web gui, navigate to Directory view, click on “Create: Directory” the disk is now available for use or we can use it for other purposes


How to: Move virtual machines between disks on Proxmox

(Note: From Proxmox VE 6.2, we can now move VM disks/Migrate VMs between local disks live (live migration), which means we can move the VM disks while the VM is still running)

This is actually an easy task.

We assume the VM we want to move only has two virtual disks, one is the system disk (Bootdisk) another is EFI Disk (Usually only UEFI BIOS will have EFI Disk, if you have selected non-UEFI BIOS for the virtual machine, just move the Bootdisk and any other data disks is enough), see example in the image

Proxmox move virtual machine
Proxmox move virtual machine

In the image, we have the Bootdisk already moved to the USB drive, now we want to move the EFI Disk.

1 Click on the EFI Disk

2 Click on the Move disk button at the top of the Proxmox web gui

3 Then select the destination

Move disk menu
Move disk menu

4 In the example we use usbxfs

5 Make necessary changes, e.g. “Check Delete source” then click on “Move disk” button

6 Wait till it’s finished, now the EFI Disk will be shown as “usbxfs:…..” instead of “rpool-folder:…”

7 Now we have successfully moved the virtual machine and it’s hard disk

8 We can now remove the old physical disk or do whatever we want with the old physical disk without destroy the virtual machine.

(If you wonder, the virtual machine configuration file are stored in the following location)

/etc/pve/nodes/pve/qemu-server
 
# e.g.
/etc/pve/nodes/pve/qemu-server/100.conf

Note: 100 is the VM ID, which you can find from Proxmox web gui


How to: Remove “You do not have a valid subscription for this server….” from Proxmox Virtual Environment 6.1-2,6.1-7,6.2-4/Proxmox VE 6.1-2,6.1-7,6.2-4 (PVE 6.1-2,6.1-7,6.2-4)

When login to Proxmox Virtual Enrivonment 6.1-2 (PVE 6.1-2), there is a pop up window

No valid subscription

You do not have a valid subscription for this server. Please visit www.proxmox.com to get a list of available options.

Proxmox No valid subscription
Proxmox No valid subscription

Note: PVE provides free version for us to use, features are almost identical to the paid version, the difference is that paid version provides support and paid update repository which is more stable than the free version.

1 Preparation

1.1 Login to your pve web gui

1.2 (Left hand side) Click on pve name under “Datacenter”

1.3 (Right hand side) Click on “>_ Shell” button

Note: You can also use any other SSH client to connect, does not have to be via PVE web gui

2 Easy method

2.1 Copy and paste following command to the terminal

sed -i.backup "s/data.status !== 'Active'/false/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service

2.2 Log out clear the browser cache or restart the browser then login again.

2.3 There will be no pop up window

3 Manual method

3.1 Change working directory

cd /usr/share/javascript/proxmox-widget-toolkit

3.2 Backup the file we will modify (“proxmoxlib.js”)

cp proxmoxlib.js proxmoxlib.js.backup

3.3 Open and edit the file (“proxmoxlib.js”)

nano proxmoxlib.js

3.4 If you are using other SSH client rather than via the PVE web gui, you can use Ctrl + W key in nano then type “if (data” to find “if (data.status !== ‘Active’) {“, if via web gui, do not use Ctrl + W, the terminal windows will be closed

if (data.status !== 'Active') {

3.5 Replace with

if (false) {

3.6 Use Ctrl + X, Y, Enter key to Save and exit nano

3.6 Restart Proxmox service

systemctl restart pveproxy.service

3.7 Log out clear the browser cache or restart the browser then login again.

3.8 There will be no pop up window


How to: Easily Delete/Remove ZFS pool (and disk from ZFS) on Proxmox VE (PVE) Make it available for other uses

1 Login to Proxmox web gui

2 Find the pool name we want to delete, here we use “test” as pool, “/dev/sdd” as the disk for example.

3 Launch Shell from web gui for the Proxmox host/cluster or via SSH

4 Use following commands to unmount and destroy the pool

umount -f /dev/sdd
zpool destroy test

5 Format the disks if necessary

fdisk /dev/sdd
g
w
 
# Explanation
format the disk /dev/sdd
Use GPT format
Write/Commit the changes 

6 Now we can bring back the Proxmox web gui, look at the /dev/sdd, it is now available for other usages.


How to: Upgrade from Proxmox 6.1 to Proxmox 6.2 (PVE)

Make sure you have backup before upgrading if you are using Proxmox free/community version

1 Edit “/etc/apt/sources.list” to add Proxmox VE No-Subscription Repository (If haven’t done so)

nano /etc/apt/sources.list

Add following lines to the bottom of the file (For no-subscription users/community/free) [1]

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

For Proxmox VE Enterprise users, add following lines instead [1]

deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

(The final version looks like following)

deb http://ftp.au.debian.org/debian buster main contrib
deb http://ftp.au.debian.org/debian buster-updates main contrib
#
# security updates
deb http://security.debian.org buster/updates main contrib
#
# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

Use, Ctrl + X, Y, Enter keys to Save and Exit

2 Open terminal from Proxmox host or via SSH

Use following command to upgrade Proxmox VE

apt update && apt dist-upgrade

(We can also login to Proxmox VE web gui, Navigate to Datacenter -> node/cluster Name -> Updates, click on “Refresh” then click on “>_ Upgrade”)

3 Restart Proxmox host

(Make sure there is no virtual machines running)

reboot

4 Now, we have Proxmox 6.2 running if the upgrade succeeded

References

[1] “Package Repositories – Proxmox VE”, Proxmox WiKi, 2020. [Online]. Available: https://pve.proxmox.com/wiki/Package_Repositories