Skip to content

Data Backup and Recovery

During the operation of HENGSHI SENSE, it is necessary to back up the data. When data issues arise, the backup data can be used for recovery, preventing data loss and corruption, and reducing losses.

This article introduces how to back up and restore data for the HENGSHI internal business database metadb, the HENGSHI internal acceleration engine, and the HENGSHI internal object storage Minio service.

Data Backup Command

Users can back up the data for metadb, engine, and minio using bin/dbbackup.sh. The backup files can be stored locally or on a remote machine.

sh
bin/dbbackup.sh -m [metadb|engine|minio]
                -l local backup root, absolute path, default ${PWD}/backup
                -k keep history backup number, default 3
                -h remote ip
                -r remote path root, default is ${backup_root}

Parameter Description:

  • -m parameter is required, indicating the type of data backup, an enumerated type, options are metadb, engine, minio.
  • -l parameter is optional, indicating the absolute path of the backup file when backing up locally.
  • -k parameter is optional, indicating the number of backup files to keep, default is 3, meaning the most recent 3 backup data will be kept.
  • -h parameter is optional, indicating the IP address for remote backup.
  • -r parameter is optional, indicating the absolute path of the backup file when backing up remotely, default is the same as the local path specified by the -l parameter.

Data Recovery Command

Users can restore data for metadb, engine, and minio using bin/dbrestore.sh. Data recovery can be done using local backup data or remote backup data.

sh
bin/dbrestore.sh -m [metadb|engine|minio]
                -l local backup root, absolute path, default ${PWD}/backup
                -t backup tar only file name
                -h remote ip
                -r remote path root, default is ${backup_root}

Parameter Description:

  • -m parameter is required, indicating the type of data backup, an enumerated type, options are metadb, engine, minio.
  • -l parameter is optional, indicating the absolute path when restoring using local backup data.
  • -h parameter is optional, indicating the IP address when restoring using remote backup data.
  • -r parameter is optional, indicating the absolute path when restoring using remote backup data, default is the same as the local absolute path specified by the -l parameter.
  • -t parameter is required, indicating the file name of the data package to be restored.

Backup and Recovery of Database metadb

The database metadb is the HENGSHI business database. It is recommended that users perform daily scheduled backups to prevent data loss and corruption.

Backup Data

Backup data is divided into local backup and remote backup based on the storage location of the backup files.

Local Backup

First, set the variable PATH_TO_BACKUP_FOLDER to store the absolute path of the backup data, then execute the backup command. At this time, a backup file named metadb_backup.xx-xx-xx_xx-xx-xx.tar.gz will be generated under $PATH_TO_BACKUP_FOLDER, and the default will keep the most recent 3 backup files. Refer to the following example.

shell
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m metadb -l $PATH_TO_BACKUP_FOLDER

Remote Backup

Remote backup requires setting two variables, REMOTE_IP and PATH_TO_BACKUP_FOLDER, where REMOTE_IP is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER is used to store the absolute path of the backup data. After executing the remote backup command, a backup file will be generated under PATH_TO_BACKUP_FOLDER on the remote device REMOTE_IP. Refer to the following example.

shell
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

Tip

Remote backup requires SSH access to the remote device, and the backup needs to be copied using scp.

Restore Data

Based on the storage address of the backup files, data recovery is divided into local data recovery and remote data recovery.

Local Data Recovery

When restoring data using local backup data, please follow the steps below.

  1. Set the variable PATH_TO_BACKUP_FOLDER to store the absolute path of the backup data.
sh
PATH_TO_BACKUP_FOLDER=/xxx
  1. Stop the HENGSHI service and remove the old database in the HENGSHI system.
sh
bin/hengshi-sense-bin stop all
mv pg_data pg_data.old
  1. Initialize and start metadb.
sh
bin/hengshi-sense-bin init metadb
bin/hengshi-sense-bin start metadb
  1. Use the locally stored backup file to restore the metadb data.
sh
bin/dbrestore.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -t metadb_backup.xx-xx-xx_xx:xx:xx.tar.gz
  1. Stop metadb, start the HENGSHI service, and complete the data recovery operation.
sh
bin/hengshi-sense-bin stop metadb
bin/hengshi-sense-bin start all

Remote Data Recovery

When the backup data is on a remote device, please first confirm whether you have SSH access to the remote device. If not, please obtain SSH access first, then follow the instructions below to perform data recovery.

  1. Set the variable REMOTE_IP for the IP address of the remote device and the variable PATH_TO_BACKUP_FOLDER for the absolute path of the backup data.
sh
PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx
  1. Stop the HENGSHI service and remove the old database in the HENGSHI system.
sh
bin/hengshi-sense-bin stop all
mv pg_data pg_data.old
  1. Initialize and start metadb.
sh
bin/hengshi-sense-bin init metadb
bin/hengshi-sense-bin start metadb
  1. Use the backup file stored on the remote device to restore the metadb data.
sh
bin/dbrestore.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -t metadb_backup.xx-xx-xx_xx:xx:xx.tar.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
  1. Stop metadb, start the HENGSHI service, and complete the data recovery operation.
sh
bin/hengshi-sense-bin stop metadb
bin/hengshi-sense-bin start all

Tip

Remote data recovery requires SSH access to the remote device, and the data needs to be copied using scp.

Backup and Recovery of the Engine

The engine is the HENGSHI internal acceleration engine. Normally, there is no need to back up engine data. When it is determined that a backup is needed, you can follow the instructions below.

Backup Engine

Backup data is divided into local backup and remote backup based on the storage location of the backup files.

Local Backup Engine

First, set the variable PATH_TO_BACKUP_FOLDER to store the absolute path of the backup data, then execute the backup command. At this time, a backup file named engine_backup.xx-xx-xx_xx-xx-xx.tar.gz will be generated under $PATH_TO_BACKUP_FOLDER, and the default will keep the most recent 3 backup files. Refer to the following example.

shell
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m engine -l $PATH_TO_BACKUP_FOLDER

Remote Backup Engine

Remote backup requires setting two variables, REMOTE_IP and PATH_TO_BACKUP_FOLDER, where REMOTE_IP is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER is used to store the absolute path of the backup data. After executing the remote backup command, a backup file will be generated under PATH_TO_BACKUP_FOLDER on the remote device REMOTE_IP. Refer to the following example.

shell
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m engine -l $PATH_TO_BACKUP_FOLDER -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

Tip

Remote backup requires SSH access to the remote device, and the backup needs to be copied using scp.

Restore Engine

Based on the storage address of the backup files, data recovery is divided into local data recovery and remote data recovery.

Local Data Recovery Engine

When restoring data using local backup data, please follow the steps below.

  1. Set the variable PATH_TO_BACKUP_FOLDER to store the absolute path of the backup data.
sh
PATH_TO_BACKUP_FOLDER=/xxx
  1. Stop the HENGSHI service and remove the old engine database in the HENGSHI system.
sh
bin/hengshi-sense-bin stop all
mv engine-cluster engine-cluster.old
  1. Initialize and start the engine.
sh
bin/hengshi-sense-bin init engine
bin/hengshi-sense-bin start engine
  1. Use the locally stored backup file to restore the engine data.
sh
bin/dbrestore.sh -m engine -l $PATH_TO_BACKUP_FOLDER -t engine_backup.xx-xx-xx_xx:xx:xx.tar.gz
  1. Stop the engine, start the HENGSHI service, and complete the data recovery operation.
sh
bin/hengshi-sense-bin stop engine
bin/hengshi-sense-bin start all

Remote Data Recovery Engine

When the backup data is on a remote device, please first confirm whether you have SSH access to the remote device. If not, please obtain SSH access first, then follow the instructions below to perform data recovery.

  1. Set the variable REMOTE_IP for the IP address of the remote device and the variable PATH_TO_BACKUP_FOLDER for the absolute path of the backup data.
sh
PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx
  1. Stop the HENGSHI service and move the old engine database in the HENGSHI system.
sh
bin/hengshi-sense-bin stop all
mv engine-cluster engine-cluster.old
  1. Initialize and start the engine.
sh
bin/hengshi-sense-bin init engine
bin/hengshi-sense-bin start engine
  1. Use the backup file stored on the remote device to restore the engine data.
sh
bin/dbrestore.sh -m engine -l $PATH_TO_BACKUP_FOLDER -t engine_backup.xx-xx-xx_xx:xx:xx.tar.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
  1. Stop the engine, start the HENGSHI service, and complete the data recovery operation.
sh
bin/hengshi-sense-bin stop engine
bin/hengshi-sense-bin start all

Tip

Remote data recovery requires SSH access to the remote device, and the data needs to be copied using scp.

Database Client Environment Variable Configuration

When using your team's metadb, there may be client version inconsistency issues during the upgrade, backup, and recovery of metadb. In this case, please first configure the EXTERNAL_DB_PG_PATH environment variable to specify the client path. Please refer to the following example for configuration. This configuration is required for each version upgrade, backup, and data recovery.

shell
## Temporarily effective, execute once for each upgrade, backup, and recovery
$ export EXTERNAL_DB_PG_PATH=/user/lib/pgsql-version/

Backup and Recovery of Object Storage Minio

The object storage Minio service is the HENGSHI internal image storage service.

Backup Data

Backup data is divided into local backup and remote backup based on the storage location of the backup files.

Local Backup

First, set the variable PATH_TO_BACKUP_FOLDER to store the absolute path of the backup data, then execute the backup command. At this time, a backup file named minio_backup.xx-xx-xx_xx-xx-xx.tar.gz will be generated under $PATH_TO_BACKUP_FOLDER, and the default will keep the most recent 3 backup files. Refer to the following example.

shell
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m minio -l $PATH_TO_BACKUP_FOLDER

Remote Backup

Remote backup requires setting two variables, REMOTE_IP and PATH_TO_BACKUP_FOLDER, where REMOTE_IP is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER is used to store the absolute path of the backup data. After executing the remote backup command, a backup file will be generated under PATH_TO_BACKUP_FOLDER on the remote device REMOTE_IP. Refer to the following example.

shell
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m minio -l $PATH_TO_BACKUP_FOLDER -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER

Tip

Remote backup requires SSH access to the remote device, and the backup needs to be copied using scp.

Restore Data

Based on the storage address of the backup files, data recovery is divided into local data recovery and remote data recovery.

Local Data Recovery

When restoring data using local backup data, please follow the steps below.

  1. Set the variable PATH_TO_BACKUP_FOLDER to store the absolute path of the backup data.
sh
PATH_TO_BACKUP_FOLDER=/xxx
  1. Stop the HENGSHI service and remove the old data storage directory in the HENGSHI system.
sh
bin/hengshi-sense-bin stop all
mv minio-data minio-data.old
  1. Use the locally stored backup file to restore the metadb data.
sh
bin/dbrestore.sh -m minio -l $PATH_TO_BACKUP_FOLDER -t minio_backup.xx-xx-xx_xx:xx:xx.tar.gz
  1. Start the HENGSHI service and complete the data recovery operation.
sh
bin/hengshi-sense-bin start all

Remote Data Recovery

When the backup data is on a remote device, please first confirm whether you have SSH access to the remote device. If not, please obtain SSH access first, then follow the instructions below to perform data recovery.

  1. Set the variable REMOTE_IP for the IP address of the remote device and the variable PATH_TO_BACKUP_FOLDER for the absolute path of the backup data.
sh
PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx
  1. Stop the HENGSHI service and remove the old database in the HENGSHI system.
sh
bin/hengshi-sense-bin stop all
mv minio-data minio-data.old
  1. Use the backup file stored on the remote device to restore the metadb data.
sh
bin/dbrestore.sh -m minio -l $PATH_TO_BACKUP_FOLDER -t metadb_backup.xx-xx-xx_xx:xx:xx.tar.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
  1. Start the HENGSHI service and complete the data recovery operation.
sh
bin/hengshi-sense-bin start all

Tip

Remote data recovery requires SSH access to the remote device, and the data needs to be copied using scp.

HENGSHI SENSE Platform User Manual