Data Backup and Recovery

During the operation of HENGSHI SENSE, it is necessary to back up data. When issues arise with the data, backup data can be used for recovery to prevent data loss and damage, thereby minimizing losses.

This article introduces how to perform backup and recovery operations for the internal business database metadb, the internal acceleration engine, and the internal object storage Minio service of HENGSHI SENSE.

Data Backup Command

Users can back up the data of metadb, engine, and minio using bin/dbbackup.sh. Backup files can be stored locally or on a remote machine.

bin/dbbackup.sh -m [metadb|engine|minio]
                -l local backup root, absolute path, default ${PWD}/backup
                -k keep history backup number, default 3
                -h remote ip
                -r remote path root, default is ${backup_root}
                -s exclude db name for backup, eg. -s db1 -s db2
                -t [sql.gz|tar.gz], default tar.gz; final target type: individual sql file gz or total folder tar gz
                -n pigz threads number

Parameter Description:

• -m Required parameter, specifies the type of data backup. It is an enumerated type with options: metadb, engine, and minio.
• -l Optional parameter, specifies the absolute path for local backup files.
• -k Optional parameter, specifies the number of backup files to retain. The default is 3, meaning the most recent 3 backups will be kept.
• -h Optional parameter, specifies the IP address for remote backup.
• -r Optional parameter, specifies the absolute path for remote backup files. By default, it is the same as the local path specified by the -l parameter.
• -s Optional parameter, skips the backup of databases with specified names. Multiple databases can be excluded by passing this parameter multiple times.
• -t Optional parameter, 'sql.gz' (recommended) performs compression during the backup process, speeding up the backup and reducing disk workspace requirements. The default 'tar.gz' is the older method, which compresses after backup, resulting in longer processing time and higher disk space requirements.
• -n Optional parameter, if pigz is detected, it will be used for compression. This parameter specifies the number of working threads for pigz. The default is 4, but higher values can be passed if the machine has a high CPU configuration.

Data Recovery Command

Users can perform data recovery for metadb, engine, and minio using bin/dbrestore.sh. Data recovery can be done using local backup data or remote backup data.

bin/dbrestore.sh -m [metadb|engine|minio]
                -l local backup root, absolute path, default ${PWD}/backup
                -t backup tar only file name
                -h remote ip
                -r remote path root, default is ${backup_root}
                -n pigz threads number

Parameter Description:

• -m This parameter is mandatory and specifies the type of data backup. It is an enumerated type with options: metadb, engine, minio.
• -l This parameter is optional and specifies the absolute path for local backup data recovery.
• -h This parameter is optional and specifies the IP address for remote backup data recovery.
• -r This parameter is optional and specifies the absolute path for remote backup data recovery. By default, it is the same as the absolute path specified by the -l parameter.
• -t This parameter is mandatory and specifies the name of the tar.gz file or the directory name generated by the new version of the backup to be restored.
• -n This parameter is optional. If pigz is detected, it will be used for decompression. This parameter specifies the number of working threads for pigz. The default is 4, but a higher value can be passed if the machine has a high CPU configuration.

Backup and Restore of Database metadb

The database metadb is the HENGSHI business database. It is recommended that users perform regular daily backups to prevent data loss and damage.

Backup Data

Backup data is divided into local backup and remote backup based on the storage location of the backup files.

Local Backup

First, set the variable PATH_TO_BACKUP_FOLDER to store the absolute path of the backup data, and then execute the backup command. At this point, a backup directory named metadb_backup.xx-xx-xx_xx-xx-xx will be generated under $PATH_TO_BACKUP_FOLDER. By default, the system will retain the latest 3 backups. Refer to the example below.

PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -t sql.gz

Remote Backup

Remote backup requires setting two variables: REMOTE_IP and PATH_TO_BACKUP_FOLDER. REMOTE_IP is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER stores the absolute path of the backup data. After executing the remote backup command, the backup file will be generated under PATH_TO_BACKUP_FOLDER on the remote device REMOTE_IP. Refer to the example below.

PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -t sql.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

Tip

When performing remote backup, you need SSH access to the remote device, and SCP is required for copying during the backup process.

Restore Data

Based on the location of the backup file, data restoration can be divided into local data restoration and remote data restoration.

Local Data Recovery

To recover data using locally backed-up files, follow the steps below:

1. Set the variable PATH_TO_BACKUP_FOLDER to the absolute path where the backup data is stored.

PATH_TO_BACKUP_FOLDER=/xxx

2. Stop the HENGSHI service and remove the old database from the HENGSHI system.

bin/hengshi-sense-bin stop all
mv pg_data pg_data.old

3. Initialize and start metadb.

bin/hengshi-sense-bin init metadb
bin/hengshi-sense-bin start metadb

4. Use the locally stored backup files to recover the metadb data.
   The -t parameter depends on whether the backup script is an old or new version, generating either a .tar.gz file or a directory. Pass the corresponding file or directory name.

bin/dbrestore.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -t metadb_backup.xx-xx-xx_xx-xx-xx[.tar.gz]

5. Stop metadb, start the HENGSHI service, and complete the data recovery process.

bin/hengshi-sense-bin stop metadb
bin/hengshi-sense-bin start all

Remote Data Recovery

When backup data is stored on a remote device, first ensure that you have SSH access to the remote device. If not, obtain SSH access before proceeding with the data recovery operations as guided below.

1. Set the IP address variable REMOTE_IP of the remote device and the absolute path variable PATH_TO_BACKUP_FOLDER where the backup data is stored.

PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx

2. Stop the HENGSHI service and remove the old database from the HENGSHI system.

bin/hengshi-sense-bin stop all
mv pg_data pg_data.old

3. Initialize and start metadb.

bin/hengshi-sense-bin init metadb
bin/hengshi-sense-bin start metadb

4. Use the backup files stored on the remote device to recover the metadb data.

bin/dbrestore.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -t metadb_backup.xx-xx-xx_xx-xx-xx[.tar.gz] -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

5. Stop metadb, start the HENGSHI service, and complete the data recovery operation.

bin/hengshi-sense-bin stop metadb
bin/hengshi-sense-bin start all

Tip

Remote data recovery requires SSH access to the remote device and uses scp for copying.

Engine Data Backup and Recovery

The engine is the internal acceleration engine of HENGSHI SENSE. Generally, there is no need to back up engine data. If backup is required, you can follow the operational guidelines below.

Backup Engine

Backup data is divided into local backup and remote backup based on the storage location of the backup files.

Local Backup Engine

First, set the variable PATH_TO_BACKUP_FOLDER to the absolute path where the backup data will be stored, and then execute the backup command. A backup directory named engine_backup.xx-xx-xx_xx-xx-xx will be generated under $PATH_TO_BACKUP_FOLDER. By default, the system retains the latest 3 backups. Refer to the example below:

PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m engine -l $PATH_TO_BACKUP_FOLDER -t sql.gz

Remote Backup Engine

Remote backup requires setting two variables: REMOTE_IP and PATH_TO_BACKUP_FOLDER. REMOTE_IP is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER stores the absolute path of the backup data. After executing the remote backup command, the backup file will be generated under PATH_TO_BACKUP_FOLDER on the remote device REMOTE_IP. Refer to the example below:

PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m engine -l $PATH_TO_BACKUP_FOLDER -t sql.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

Tip

When performing remote backup, SSH access to the remote device is required, and SCP must be used for copying during the backup process.

Restore Engine

Based on the location where the backup file is stored, it can be divided into local data recovery and remote data recovery.

Local Data Recovery Engine

When recovering data using locally backed-up files, follow the steps below:

1. Set the variable PATH_TO_BACKUP_FOLDER to the absolute path where the backup data is stored.

PATH_TO_BACKUP_FOLDER=/xxx

2. Stop the HENGSHI service and remove the old database from the HENGSHI system.

bin/hengshi-sense-bin stop all
mv engine-cluster engine-cluster.old

3. Initialize and start the engine.

bin/hengshi-sense-bin init engine
bin/hengshi-sense-bin start engine

4. Use the locally stored backup files to recover the engine's data.
   The -t parameter depends on whether the backup script is an old or new version, generating either a .tar.gz file or a directory. Pass the corresponding file or directory name.

bin/dbrestore.sh -m engine -l $PATH_TO_BACKUP_FOLDER -t engine_backup.xx-xx-xx_xx-xx-xx[.tar.gz]

5. Stop the engine, start the HENGSHI service, and complete the data recovery process.

bin/hengshi-sense-bin stop engine
bin/hengshi-sense-bin start all

Remote Data Recovery Engine

When backup data is stored on a remote device, first confirm whether you have SSH access to the remote device. If not, obtain SSH access first, and then follow the instructions below to perform data recovery operations.

1. Set the IP address variable REMOTE_IP of the remote device and the absolute path variable PATH_TO_BACKUP_FOLDER where the backup data is stored.

PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx

2. Stop the HENGSHI service and move the old engine database in the HENGSHI system.

bin/hengshi-sense-bin stop all
mv engine-cluster engine-cluster.old

3. Initialize and start the engine.

bin/hengshi-sense-bin init engine
bin/hengshi-sense-bin start engine

4. Use the backup files stored on the remote device to perform data recovery for the engine.

bin/dbrestore.sh -m engine -l $PATH_TO_BACKUP_FOLDER -t engine_backup.xx-xx-xx_xx-xx-xx[.tar.gz] -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

5. Stop the engine, start the HENGSHI service, and complete the data recovery operation.

bin/hengshi-sense-bin stop engine
bin/hengshi-sense-bin start all

Tip

Remote data recovery requires SSH access to the remote device and uses scp for copying.

Database Client Environment Variable Configuration

When using your team's metadb, issues with client version inconsistency may occur during metadb upgrades, backups, and restores. In such cases, please configure the EXTERNAL_DB_PG_PATH environment variable to specify the client path. Refer to the example below for configuration. This configuration needs to be performed for each version upgrade, backup, and data restore.

## Temporarily effective; must be executed once for each upgrade, backup, and restore
$ export EXTERNAL_DB_PG_PATH=/user/lib/pgsql-version/

Backup and Restore of Object Storage Minio

The object storage Minio service is the internal image storage service of HENGSHI SENSE.

Backup Data

Backup data is divided into local backup and remote backup based on the storage location of the backup files.

Local Backup

First, set the variable PATH_TO_BACKUP_FOLDER to store the absolute path of the backup data, and then execute the backup command. At this point, a backup file named minio_backup.xx-xx-xx_xx-xx-xx.tar.gz will be generated under $PATH_TO_BACKUP_FOLDER. By default, the system will retain the most recent 3 backup files. Refer to the example below.

PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m minio -l $PATH_TO_BACKUP_FOLDER

Remote Backup

Remote backup requires setting two variables: REMOTE_IP and PATH_TO_BACKUP_FOLDER. REMOTE_IP is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER stores the absolute path of the backup data. After executing the remote backup command, the backup file will be generated under PATH_TO_BACKUP_FOLDER on the remote device REMOTE_IP. Refer to the example below.

PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m minio -l $PATH_TO_BACKUP_FOLDER -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

Tip

When performing remote backup, you need SSH access to the remote device, and SCP is required for copying during the backup process.

Restore Data

Based on the location where the backup file is stored, data restoration can be divided into local data restoration and remote data restoration.

Local Data Recovery

When recovering data using locally backed-up files, follow the steps below:

1. Set the variable PATH_TO_BACKUP_FOLDER to the absolute path where the backup data is stored.

PATH_TO_BACKUP_FOLDER=/xxx

2. Stop the HENGSHI service and remove the old data storage directory from the HENGSHI system.

bin/hengshi-sense-bin stop all
mv minio-data minio-data.old

3. Use the locally stored backup file to restore the MinIO data.

bin/dbrestore.sh -m minio -l $PATH_TO_BACKUP_FOLDER -t minio_backup.xx-xx-xx_xx-xx-xx.tar.gz

4. Start the HENGSHI service to complete the data recovery process.

bin/hengshi-sense-bin start all

Remote Data Recovery

When backup data is stored on a remote device, first ensure that you have SSH access to the remote device. If not, obtain SSH access first, and then follow the instructions below to perform data recovery.

1. Set the IP address variable REMOTE_IP of the remote device and the absolute path variable PATH_TO_BACKUP_FOLDER where the backup data is stored.

PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx

2. Stop the HENGSHI service and remove the old database from the HENGSHI system.

bin/hengshi-sense-bin stop all
mv minio-data minio-data.old

3. Use the backup files stored on the remote device to restore the MinIO data.

bin/dbrestore.sh -m minio -l $PATH_TO_BACKUP_FOLDER -t minio_backup.xx-xx-xx_xx-xx-xx.tar.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

4. Start the HENGSHI service to complete the data recovery process.

bin/hengshi-sense-bin start all

Tip

Remote data recovery requires SSH access to the remote device and uses SCP for copying.