Backup system: general information

A new backup system is launched in ISPmanager Lite and Business starting from version 5.52.0. This new system is based on ISPtar.

In ISPmanager Lite it will change the DAR-based system.

In ISPmanager Business this system will allow an administrator to choose between a simplified version and advanced configuration options and filters.

New features

The main differences from the old backup system:

• A separate backup copy is created for each user.
• Filters are not used. A server administrator can exclude file copies, directories, databases (by default, all user data are backed up), or exclude a certain user from the backup.
• Only one storage can be used, however, you can easily switch between storages. In this case, you won't be able to use backups on the first storage and can access backups on the second storage.
• By default, backups are made once a day. The first weekly backup copy (the week starts from Sunday) is Full, other copies — Differential. You can change the backup period in the Cron configuration form.
• You can set limits on total backup size. Once the limit is reached, the system will delete the oldest copies, and store the equal number of daily and weekly backups, if possible.
• A backup copy of a disabled user will be made only once after its suspension.
• If the total backup size is not limited, and there is no free disk space on the storage, the system will delete the oldest backups.
• The total number of backups that are kept in the system: 7 full weekly backups and 7 differential daily backups. You can change the number with the BackupCountLimit parameter in the etc/ispmgr.conf configuration file.

In version 5.137 you can specify the number of full and daily (differential) backup copies:

BackupCountLimit 7:7

The minimum value for the BackupCountLimit parameter is 2:2. It means that 2 full and 2 differential copiers will be made.

Before version 5.137 you should specify the parameter as follows:

BackupCountLimit 8

The minimum value for the BackupCountLimit parameter is 2. It means that 1 full and 1 differential copiers will be made. .

• Users can now download their backups, and upload them to the server.

ISPtar delivers a number of advantages:

• A backup copy is divided into small volumes (20 MB in size by default) allowing to reduce recovery time in case of partial data restore, and amount of free disk space to create and restore a backup copy.
• (will be launched in version 5.55) There is no need to compress some types of files. This enables to reduce CPU usage during backup. Disable file compression via the etc/isptar.conf file.
• * You can extract data from backup without ISPmanager via the console.

Switching from old backup system

When you update ISPmanager Lite to new version, a backup module based on DAR will be changed into ISPtar: the same storage will be connected, old backup copies won't be accessible from the web-interface (files are not deleted from storage), they can be converted to a new format by running /usr/local/mgr5/sbin/backup2_conv (this process may take a long time. Make sure you have enough user disk quota to store files after they are converted from DAR into ISPtar).

If you connected the DAR storage and started the conversion process, but neither archives nor errors appeared in var/backup2.conv, you should copy all the information files from the storage into a temporary directory of the control panel, and start the conversion again:

Note: if you import backup copies into a new server, make sure that dar is set up on that server

cd /usr/local/mgr5
cp -pr /<DARstorage>/201*dar var/backup/ispmgr/
cp -pr /<DARstorage>/*info var/backup/ispmgr/
sbin/backup2_conv &

ISPtar does not conflict with the oldest backup module (the one activated with the EnableOldBackup option), and both systems can be used in parallel.

Module configuration

Click Settings in the Backups module to select and configure a storage, set the total number of backups to store, files (you can use regular expressions) and databases that you want to exclude from the backup.

You can use the nice and ionice utilities to set priorities. Add the BackupCommandPrefix parameter in the etc/ispmgr.conf configuration file to set priorities. nice -n 10 ionice -c2 -n7 is set by default.

Storage size limit

This field enables to set the maximum size for backup copies on storage (if a local storage is used on ISPmanager Business, the limit will be applied to every cluster node). When connecting a local storage for the first time, starting from version 5.63, the limit of 50% of free disk space will be set. This value can be modified. Starting from version 5.63 the system includes protection from local storage overload: when uploading data to local storage, the system will check the disk, and if less than 5% of disk space is left, will try to delete old archives. If that attempt fails, the backup process will finish with the following error:

backup DEBUG backup2_storage.cpp:119 Available limit with restriction 95 pct '0' mib
backup DEBUG backup2_storage.cpp:120 File size '54' mib
main DEBUG backup2_cp2_server.cpp:354 m_read = 'RELEASE 56686365
main ERROR Upload failed
libmgr ERROR Error: Type: 'Cant split slices' Object: ' ' Value: 'Part name prefix missed for part ' '

Storage scanning

Storage scanning is performed when connecting to a new storage type or when switching to a new directory in the storage.

From version 5.58 the first backup forms a full copy after the storage directory has been changed, regardless full backup copies made in previous days.

User's storage

From version, 5.58 users of ISPmanager Business have an opportunity to set up their own remote storage (using the same user disk space (via FTP/SFTP) is not recommended.) It is possible to set up disk space limit, but database/files exceptions, which administrator have set before, will be applied (because one copy is located in administrator storage, then it is copied to user storage).

If administrator disables backups for a certain user, anyway this user will be able to set his own storage, and backups will be created automatically and scheduled in his storage. Furthermore, if administrator disables backups, the user still can activate his storage and create copies, restore data, and import archives manually.

But, if the administrator enables user backups and the backup process fails, the backup copy won't be added into the user storage. In this case, the user can ask his administrator to disable backups in the user edit form.

View old file version

To view contents of an old file, download the file from the backup copy.

Full user recovery

To restore user data from a backup, enter Tools → Backup copies → select the backup→ Details button → select the user → Restore button → OK . When the data are restored, the message "Backup restore has been completed successfully" will appear in the ISPmanager interface.

Excluding files from a backup copy

Go to the Backups module to choose files you want to exclude from the backup. Enter paths relative to a user's home directory, e.g. data/.filemgr-tmp. In filters, you can use regular expressions (*). The files that you select won't be backed up.

In the field below you can exclude databases (you should enter a full database name in a separate line).

Changing the backup start time

By default, backups are made once a day. In the Scheduler module, you can change this period for the ISPmanager backup task.

Setting backup time

The BackupTimeInterval parameter enables to set backup time. E.g.: BackupTimeInterval 03:00:00-06:00:00 means that the backup process will start from 3 to 6 a.m. The process for a separate user won't be interrupted. If the system doesn't have enough time to back up all users within that period, the process will continue the next day. New copies for that day won't be created.

E.g.: on Monday, September 14, 2015, we backed up 10 users from 15, and the remaining 5 users will be backed up on Tuesday. The next backup copy for all users will be created on Wednesday, September 16, 2015. User backups will be dated 14.09.2015, although some users were backed up only on September 15, 2015.

Clearing disk space in the storage

You can clear disk space by deleting backup copies you don't need from a list of backups.

You can also set the Total size in the Backups module. Once you are done, old backups will be deleted until the total size reaches the specified value.

If you run ISPmanager Business with a local storage, and there are multiple nodes for backups, this size will be applied to each node. Click the "Location" button to view the distribution of backup copies on cluster nodes.

Speeding up the backup process

If you want to make a large backup copy, you can change the priority with the BackupCommandPrefix parameter in the etc/ispmgr.conf configuration file. nice -n 10 ionice -c2 -n7. is set by default.

The nice utility invokes a utility, requesting that it be run with a different system scheduling priority. More details can be found here

ionice is used to get and set the class and process priority.

Service files of a backup copy

Once the backup copy has been made, the files 2015-08-12.user.tgz and 2015-08-12.user.info will be created in /usr/local/mgr5/var/backup/ispmgr. They keep service information and do not contain the backup itself.

The tgz file contains information about backup files.

The info file contains information about its creation date, the name of the last backup file, backup size, and type.

Such files make it possible to show backup copies in the list and reduce the time required to access data.

Limit on the number of backup copies

The number of backup copies to be saved per user is set with the BackupCountLimit parameter in etc/ispmgr.conf.

Odd number of backup copies

If the BackupCountLimit parameter is an odd number, for example, 15, 8 full backups and 7 daily incremental backups will be created. When the limit is exceeded, a daily incremental backup will be deleted first.

I.e. the system will try to create an equal number of full and differential backups. However, there will be more full backup copies (+1).

Objects excluded from backup

Due to technical reasons, email filters cannot be backed up.

How often backup can be made

A user can't make a backup copy more often than once an hour, otherwise, a previously cached backup copy will be downloaded.

A backup copy of a removed user

After deleting a user whose data were backed up, the "!" icon will be displayed in the list of saved data. This icon contains the date when that user was deleted. It won't be deleted upon user restore.

Correspondence between full and daily backup copies

When the backup process starts for the first time, a full backup copy will be made. Next days (except for Sunday) daily differential copies will be made, and on Sunday a full backup will be created. During the first 14 days, 3 full and 11 differential backups will be made. When 14 days are passed, the oldest differential copies will be deleted first. Once a full copy doesn't have any incremental parts, it will be deleted. Thus, 7 full and 7 differential copies will be left in about 2 months.

Deleting backups

Automatically

Old, non-blocked backup copies will be deleted when

• storage disk size reaches its limit,
• when disk space limit reaches 95% of its total size (for local storage),
• when a user's backup limit is reached (BackupCountLimit), the oldest copy will be deleted.

The last full archive, today's archive, and the archive uploaded by the user (custom) won't be deleted automatically.

Manually

An administrator can delete backups in the list of backup copies (the "Delete" button). In ISPmanager Business a user can delete backups in the custom storage.

In order to delete a backup copy, in the /usr/local/mgr5 directory you need to specify the BACKUP_TOKEN environment variable, execute the sbin/backup2_cp --delete command, and specify a path to the info file of the backup copy. The token value is kept in etc/ispmgr.conf in the BackupToken parameter.

A storage type and token are stored in etc/ispmgr.conf, the BackupType and BackupToken parameters correspondingly.

The following example shows deletion of the backup copy of the certain user (username) from the local storage:

BACKUP_TOKEN="type=local;url=/var/backups/";sbin/backup2_cp --delete var/backup/ispmgr/2015-09-08.username.info

The following example shows deletion of all copies of the certain user (username) from the FTP storage:

BACKUP_TOKEN="password=qwerty12345;type=ftp;url=[ftp://backup5.reserv.net;username=ftpuser23|ftp://backup5.reserv.net;username=ftpuser23] "; ls -1 var/backup/ispmgr/*username.info | xargs -I {} sbin/backup2_cp --delete {}

sbin/backup2_cp --delete var/backup/ispmgr/2015-09-08.username.info --type local --token /var/backup

The following example shows deletion of all copies in the local storage:

BACKUP_TOKEN="type=local;url=/var/backups/"; ls -1 var/backup/ispmgr/*.info | xargs -I {} sbin/backup2_cp --delete {}

Changing volume size

You can change the volume size in the etc/ispmgr.conf configuration file.

Example:

BackupSliceSize 30M

Restoring/downloading file/database

An administrator cannot restore files. You should log in as a user (in the "Backups" module click the "Enter" button) into the backup copy, select a file(s)/database(s), and click "Restore"/"Download".

Starting from ISPmanager Business 5.68.0 (released on 09.08.2016) users can restore/download separate mailboxes.

Reserving system data

/usr/local/mgr5/etc, /usr/local/mgr5/var/userconf and /etc will be archived into archives like F2015-09-16.root.tgz in ISPmanager 5 Lite and F2016-06-11.root#system_localhost.tgz and F2016-06-11.root#system_.tgz in ISPmanager Business.

Starting from ISPmanager Business 5.59 the system backup copy will also include a mysql databse of the ispmgr master-server. Database dump is made in default encoding and is put into archive like F2016-06-11.root#.tgz.

These archives are not available from the panel interface, as they are used very seldom.

In order to restore data, go to the storage, e.g. /var/backup. Select a file of the root user archive (usually, root) made on a required date. Execute the commands to view the archive:

isptar -l F2015-09-16.root.tgz
tar -tf F2015-09-16.root.tgz

Example of extraction of the ispmgr.root.dashboard.xml file:

tar -zxvf F2015-09-16.root.tgz usr/local/mgr5/var/userconf/ispmgr.root.dashboard.xml

How much disk space is required for backups

On the server

The panel's temporary directory: by default /usr/local/mgr5/tmp The panel's system directory: by default /usr/local/mgr5/var/backup/ispmgr The size of one archive part: by default 100Mb, it can be changed through the option BackupSliceSize

If a non-local storage is used:

In user disk quota

You use should have enough disk space for at least one part of backup (by default — 100 MB), because activation is made with user permissions and a part of archive belongs to the user due to security reasons.

Changing temporary directory

If you need to change a temporary directory, copy the /usr/local/mgr5/var/backup/ispmgr directory and mount a required section in /usr/local/mgr5/var/backup/ispmgr.

mount --bind /required/section /usr/local/mgr5/var/backup/ispmgr

If you change a path to the temporary directory, the backup process may cause issues. We don't recommend using the parameter described below:

You can add the string into the /usr/local/mgr5/etc/ispmgr.conf configuration file to change a path to the temporary directory

path BackupLocalDir <path>

Execute killall core

IF the killall utility is not present, execute

/usr/local/mgr5/sbin/mgrctl -m ispmgr exit

Enabling detailed logging

To enable detailed logging, add the following strings into etc/debug.conf:

backup2.*       9
backup2_import.*        9
backup2_download.*      9
backup2_cp.*    9
restore2.*      9
backup2_cgi.*   9
backup2_conv.*   9
backup2_system.*       9

and execute killall core

You can also enable output of all backup logs:

tail -f /usr/local/mgr5/var/backup2*log /usr/local/mgr5/var/restore2.log

Making a backup copy manually

For the current date

1) business:

cd /usr/local/mgr5 && ./sbin/backup2_pro &

2) lite:

cd /usr/local/mgr5 && ./sbin/backup2 &

For a specific date

Be careful when providing a date, which is too far from the current date (large than BackupCountLimit/2 weeks)! If you make a copy in the future and the other one with the date, which exceeds the previous date, all the copies might be deleted by the system that checks backup size (even if the future backup copy doesn't contain users).

We do not recommend applying the method described unless it is strictly necessary. Be ready for unexpected results!

1) business:

cd /usr/local/mgr5 && ./sbin/backup2_pro --date 2016-05-01 &

2) lite:

cd /usr/local/mgr5 && ./sbin/backup2 --date 2016-05-01 &

For a specific date and user

Be careful when providing a date, which is too far from the current date (large than BackupCountLimit/2 weeks)! If you make a copy in the future and the other one with the date, which exceeds the previous date, all the copies might be deleted by the system that checks backup size (even if the future backup copy doesn't contain users).

We do not recommend applying the method described unless it is strictly necessary. Be ready for unexpected results!

1) business:

cd /usr/local/mgr5 && ./sbin/backup2_pro --date 2016-05-01 user1 &

2) lite:

cd /usr/local/mgr5 && ./sbin/backup2 --date 2016-05-01 user1 &

Extracting data

If you cannot extract data from the control panel, but you have the following files in the storage

F2016-11-02.user.tgz.part1
F2016-11-02.user.tgz.part2

you can add them to achieve by executing the command:

Unix (Linux/FreeBSD/MacOS):

cat F2016-11-02.user.tgz.part1 F2016-11-02.user.tgz.part2  > F2016-11-02.user.tgz

Windows:

copy /b F2016-11-02.user.tgz.part1 + /b F2016-11-02.user.tgz.part2 F2016-11-02.user.tgz

Then you will be able to extract data from the F2016-11-02.user.tgz archive using the tools provided by your operating system, but its import into ISPmanager is not possible.

FAQ

The system deletes old backups

Sometimes old backup copies may be deleted from the storage. If an FTP-storage is used and the system gets the timeout after 15 minutes, the backup system considers the storage to be full and deletes the least important old copy.

User files are not included in the backup

If user directories are connected as separate partitions, their files from the home directory (eg. /var/www/user/data/www/*) won't be included into a backup copy.

GoogleDrive. Delete the directory that the backup to cart is connected

If you need to delete the directory in Google Drive storage, you first need to disable the backup function in ISPmanager, otherwise, it may lead to unexpected results.