Data Backup and Recovery
During the operation of HENGSHI SENSE, it is necessary to back up the data. When data issues arise, the backup data can be used for recovery, preventing data loss and corruption, and reducing losses.
This article introduces how to back up and restore data for the HENGSHI internal business database metadb, the HENGSHI internal acceleration engine, and the HENGSHI internal object storage Minio service.
Data Backup Command
Users can back up the data for metadb, engine, and minio using bin/dbbackup.sh
. The backup files can be stored locally or on a remote machine.
bin/dbbackup.sh -m [metadb|engine|minio]
-l local backup root, absolute path, default ${PWD}/backup
-k keep history backup number, default 3
-h remote ip
-r remote path root, default is ${backup_root}
Parameter Description:
-m
parameter is required, indicating the type of data backup, an enumerated type, options are metadb, engine, minio.-l
parameter is optional, indicating the absolute path of the backup file when backing up locally.-k
parameter is optional, indicating the number of backup files to keep, default is 3, meaning the most recent 3 backup data will be kept.-h
parameter is optional, indicating the IP address for remote backup.-r
parameter is optional, indicating the absolute path of the backup file when backing up remotely, default is the same as the local path specified by the-l
parameter.
Data Recovery Command
Users can restore data for metadb, engine, and minio using bin/dbrestore.sh
. Data recovery can be done using local backup data or remote backup data.
bin/dbrestore.sh -m [metadb|engine|minio]
-l local backup root, absolute path, default ${PWD}/backup
-t backup tar only file name
-h remote ip
-r remote path root, default is ${backup_root}
Parameter Description:
-m
parameter is required, indicating the type of data backup, an enumerated type, options are metadb, engine, minio.-l
parameter is optional, indicating the absolute path when restoring using local backup data.-h
parameter is optional, indicating the IP address when restoring using remote backup data.-r
parameter is optional, indicating the absolute path when restoring using remote backup data, default is the same as the local absolute path specified by the-l
parameter.-t
parameter is required, indicating the file name of the data package to be restored.
Backup and Recovery of Database metadb
The database metadb is the HENGSHI business database. It is recommended that users perform daily scheduled backups to prevent data loss and corruption.
Backup Data
Backup data is divided into local backup and remote backup based on the storage location of the backup files.
Local Backup
First, set the variable PATH_TO_BACKUP_FOLDER
to store the absolute path of the backup data, then execute the backup command. At this time, a backup file named metadb_backup.xx-xx-xx_xx-xx-xx.tar.gz
will be generated under $PATH_TO_BACKUP_FOLDER
, and the default will keep the most recent 3 backup files. Refer to the following example.
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m metadb -l $PATH_TO_BACKUP_FOLDER
Remote Backup
Remote backup requires setting two variables, REMOTE_IP
and PATH_TO_BACKUP_FOLDER
, where REMOTE_IP
is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER
is used to store the absolute path of the backup data. After executing the remote backup command, a backup file will be generated under PATH_TO_BACKUP_FOLDER
on the remote device REMOTE_IP
. Refer to the following example.
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
Tip
Remote backup requires SSH access to the remote device, and the backup needs to be copied using scp
.
Restore Data
Based on the storage address of the backup files, data recovery is divided into local data recovery and remote data recovery.
Local Data Recovery
When restoring data using local backup data, please follow the steps below.
- Set the variable
PATH_TO_BACKUP_FOLDER
to store the absolute path of the backup data.
PATH_TO_BACKUP_FOLDER=/xxx
- Stop the HENGSHI service and remove the old database in the HENGSHI system.
bin/hengshi-sense-bin stop all
mv pg_data pg_data.old
- Initialize and start metadb.
bin/hengshi-sense-bin init metadb
bin/hengshi-sense-bin start metadb
- Use the locally stored backup file to restore the metadb data.
bin/dbrestore.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -t metadb_backup.xx-xx-xx_xx:xx:xx.tar.gz
- Stop metadb, start the HENGSHI service, and complete the data recovery operation.
bin/hengshi-sense-bin stop metadb
bin/hengshi-sense-bin start all
Remote Data Recovery
When the backup data is on a remote device, please first confirm whether you have SSH access to the remote device. If not, please obtain SSH access first, then follow the instructions below to perform data recovery.
- Set the variable
REMOTE_IP
for the IP address of the remote device and the variablePATH_TO_BACKUP_FOLDER
for the absolute path of the backup data.
PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx
- Stop the HENGSHI service and remove the old database in the HENGSHI system.
bin/hengshi-sense-bin stop all
mv pg_data pg_data.old
- Initialize and start metadb.
bin/hengshi-sense-bin init metadb
bin/hengshi-sense-bin start metadb
- Use the backup file stored on the remote device to restore the metadb data.
bin/dbrestore.sh -m metadb -l $PATH_TO_BACKUP_FOLDER -t metadb_backup.xx-xx-xx_xx:xx:xx.tar.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
- Stop metadb, start the HENGSHI service, and complete the data recovery operation.
bin/hengshi-sense-bin stop metadb
bin/hengshi-sense-bin start all
Tip
Remote data recovery requires SSH access to the remote device, and the data needs to be copied using scp
.
Backup and Recovery of the Engine
The engine is the HENGSHI internal acceleration engine. Normally, there is no need to back up engine data. When it is determined that a backup is needed, you can follow the instructions below.
Backup Engine
Backup data is divided into local backup and remote backup based on the storage location of the backup files.
Local Backup Engine
First, set the variable PATH_TO_BACKUP_FOLDER
to store the absolute path of the backup data, then execute the backup command. At this time, a backup file named engine_backup.xx-xx-xx_xx-xx-xx.tar.gz
will be generated under $PATH_TO_BACKUP_FOLDER
, and the default will keep the most recent 3 backup files. Refer to the following example.
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m engine -l $PATH_TO_BACKUP_FOLDER
Remote Backup Engine
Remote backup requires setting two variables, REMOTE_IP
and PATH_TO_BACKUP_FOLDER
, where REMOTE_IP
is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER
is used to store the absolute path of the backup data. After executing the remote backup command, a backup file will be generated under PATH_TO_BACKUP_FOLDER
on the remote device REMOTE_IP
. Refer to the following example.
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m engine -l $PATH_TO_BACKUP_FOLDER -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
Tip
Remote backup requires SSH access to the remote device, and the backup needs to be copied using scp
.
Restore Engine
Based on the storage address of the backup files, data recovery is divided into local data recovery and remote data recovery.
Local Data Recovery Engine
When restoring data using local backup data, please follow the steps below.
- Set the variable
PATH_TO_BACKUP_FOLDER
to store the absolute path of the backup data.
PATH_TO_BACKUP_FOLDER=/xxx
- Stop the HENGSHI service and remove the old engine database in the HENGSHI system.
bin/hengshi-sense-bin stop all
mv engine-cluster engine-cluster.old
- Initialize and start the engine.
bin/hengshi-sense-bin init engine
bin/hengshi-sense-bin start engine
- Use the locally stored backup file to restore the engine data.
bin/dbrestore.sh -m engine -l $PATH_TO_BACKUP_FOLDER -t engine_backup.xx-xx-xx_xx:xx:xx.tar.gz
- Stop the engine, start the HENGSHI service, and complete the data recovery operation.
bin/hengshi-sense-bin stop engine
bin/hengshi-sense-bin start all
Remote Data Recovery Engine
When the backup data is on a remote device, please first confirm whether you have SSH access to the remote device. If not, please obtain SSH access first, then follow the instructions below to perform data recovery.
- Set the variable
REMOTE_IP
for the IP address of the remote device and the variablePATH_TO_BACKUP_FOLDER
for the absolute path of the backup data.
PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx
- Stop the HENGSHI service and move the old engine database in the HENGSHI system.
bin/hengshi-sense-bin stop all
mv engine-cluster engine-cluster.old
- Initialize and start the engine.
bin/hengshi-sense-bin init engine
bin/hengshi-sense-bin start engine
- Use the backup file stored on the remote device to restore the engine data.
bin/dbrestore.sh -m engine -l $PATH_TO_BACKUP_FOLDER -t engine_backup.xx-xx-xx_xx:xx:xx.tar.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
- Stop the engine, start the HENGSHI service, and complete the data recovery operation.
bin/hengshi-sense-bin stop engine
bin/hengshi-sense-bin start all
Tip
Remote data recovery requires SSH access to the remote device, and the data needs to be copied using scp
.
Database Client Environment Variable Configuration
When using your team's metadb, there may be client version inconsistency issues during the upgrade, backup, and recovery of metadb. In this case, please first configure the EXTERNAL_DB_PG_PATH
environment variable to specify the client path. Please refer to the following example for configuration. This configuration is required for each version upgrade, backup, and data recovery.
## Temporarily effective, execute once for each upgrade, backup, and recovery
$ export EXTERNAL_DB_PG_PATH=/user/lib/pgsql-version/
Backup and Recovery of Object Storage Minio
The object storage Minio service is the HENGSHI internal image storage service.
Backup Data
Backup data is divided into local backup and remote backup based on the storage location of the backup files.
Local Backup
First, set the variable PATH_TO_BACKUP_FOLDER
to store the absolute path of the backup data, then execute the backup command. At this time, a backup file named minio_backup.xx-xx-xx_xx-xx-xx.tar.gz
will be generated under $PATH_TO_BACKUP_FOLDER
, and the default will keep the most recent 3 backup files. Refer to the following example.
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m minio -l $PATH_TO_BACKUP_FOLDER
Remote Backup
Remote backup requires setting two variables, REMOTE_IP
and PATH_TO_BACKUP_FOLDER
, where REMOTE_IP
is used to store the IP address of the remote device, and PATH_TO_BACKUP_FOLDER
is used to store the absolute path of the backup data. After executing the remote backup command, a backup file will be generated under PATH_TO_BACKUP_FOLDER
on the remote device REMOTE_IP
. Refer to the following example.
PATH_TO_BACKUP_FOLDER=/xxx
bin/dbbackup.sh -m minio -l $PATH_TO_BACKUP_FOLDER -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
Tip
Remote backup requires SSH access to the remote device, and the backup needs to be copied using scp
.
Restore Data
Based on the storage address of the backup files, data recovery is divided into local data recovery and remote data recovery.
Local Data Recovery
When restoring data using local backup data, please follow the steps below.
- Set the variable
PATH_TO_BACKUP_FOLDER
to store the absolute path of the backup data.
PATH_TO_BACKUP_FOLDER=/xxx
- Stop the HENGSHI service and remove the old data storage directory in the HENGSHI system.
bin/hengshi-sense-bin stop all
mv minio-data minio-data.old
- Use the locally stored backup file to restore the metadb data.
bin/dbrestore.sh -m minio -l $PATH_TO_BACKUP_FOLDER -t minio_backup.xx-xx-xx_xx:xx:xx.tar.gz
- Start the HENGSHI service and complete the data recovery operation.
bin/hengshi-sense-bin start all
Remote Data Recovery
When the backup data is on a remote device, please first confirm whether you have SSH access to the remote device. If not, please obtain SSH access first, then follow the instructions below to perform data recovery.
- Set the variable
REMOTE_IP
for the IP address of the remote device and the variablePATH_TO_BACKUP_FOLDER
for the absolute path of the backup data.
PATH_TO_BACKUP_FOLDER=/xxx
REMOTE_IP=/xxx
- Stop the HENGSHI service and remove the old database in the HENGSHI system.
bin/hengshi-sense-bin stop all
mv minio-data minio-data.old
- Use the backup file stored on the remote device to restore the metadb data.
bin/dbrestore.sh -m minio -l $PATH_TO_BACKUP_FOLDER -t metadb_backup.xx-xx-xx_xx:xx:xx.tar.gz -h $REMOTE_IP -r $PATH_TO_BACKUP_FOLDER
- Start the HENGSHI service and complete the data recovery operation.
bin/hengshi-sense-bin start all
Tip
Remote data recovery requires SSH access to the remote device, and the data needs to be copied using scp
.