Container Deployment

This article explains how to deploy HENGSHI SENSE on a container, as well as how to perform version upgrades and data backups after deployment.

Pre-deployment Preparation

Before single-machine deployment, please complete the following preparations.

1. Check the Docker environment. The requirements are:
   • Docker version >= 17.09
   • Docker-compose installed.

Refer to the documentation Offline Installation of Docker and Docker-compose.

2. Obtain and import the offline image.

wget https://download.hengshi.com/releases/hengshi-sense-xxx.tar.gz
docker load -i hengshi-sense-xxx.tar.gz

3. Obtain the docker-compose deployment file based on the version.

Installation Version	Deployment File	Component Dependencies
5.1.x	docker-compose	metadb, engine, hengshi, minio, redis, flink (removed, no need to run), apm-server
5.3.x	docker-compose	metadb, engine, hengshi, minio, redis, apm-server
5.4.x	docker-compose	metadb, engine, hengshi, minio, redis, apm-server

Standalone Deployment

Follow the steps below to deploy the HENGSHI service in standalone mode.

1. Extract the deployment package

# Extract the downloaded docker-compose file
unzip -q docker-compose-x.x.zip
# Navigate to the configuration file directory, e.g., if the downloaded version is 5.1, then cd docker-compose-5.1
cd docker-compose-x.x

2. Modify the configuration

Refer to the detailed variables in .env

# Rename the default configuration to .env and modify the variables in .env
# HS_IMAGE_TAG must be updated to the tag of the imported offline image
cp hsenv .env

Note

When configuring .env, ensure that the configured ports are not occupied. You can use the netstat -anpt command to check.

If the built-in engine type requires Doris, adjust the HS_ENGINE_TYPE parameter.

Uncomment the "networks" section in the docker-compose file to prevent Doris from failing to start due to IP changes.

3. Initialize metadb and engine

# Before installation, ensure that metadb and engine initialization operations are performed; otherwise, HENGSHI cannot start
docker-compose run --rm metadb init metadb # Initialize metadb
docker-compose run --rm engine init engine # Initialize engine

If the operation is successful, the following messages will appear.

docker-compose run --rm metadb init metadb
# If the following message appears in the logs, it indicates that metadb initialization was successful
INIT SUCCESS! : [metadb] /opt/hsdata/pg_data

docker-compose run --rm engine init engine
# If the following message appears in the logs, it indicates that engine initialization was successful
INIT SUCCESS! : [engine] /opt/hsdata/engine-cluster
after config docker engine standalone

4. Start the service

docker-compose up -d  # Start in detached mode

If the operation is successful, the following messages will appear.

docker-compose up -d
[+] Running 8/8
 ✔ Container hengshi-flink      Started   1.1s
 ✔ Container hengshi-metadb     Started   1.1s
 ✔ Container hengshi-engine     Started   1.0s
 ✔ Container hengshi-apmserver  Started   0.8s
 ✔ Container hengshi-minio      Started   1.0s
 ✔ Container hengshi-monit      Started   1.0s
 ✔ Container hengshi-redis      Started   1.2s
 ✔ Container hengshi-sense      Started   1.9s

5. Stop the service and remove containers

docker-compose down # Destroy containers

If the operation is successful, the following messages will appear.

docker-compose down
[+] Running 9/8
 ✔ Container hengshi-monit      Removed    0.0s
 ✔ Container hengshi-apmserver  Removed    0.0s
 ✔ Container hengshi-engine     Removed   10.4s
 ✔ Container hengshi-sense      Removed    0.0s
 ✔ Container hengshi-flink      Removed   10.3s
 ✔ Container hengshi-metadb     Removed   10.4s
 ✔ Container hengshi-minio      Removed   10.4s
 ✔ Container hengshi-redis      Removed   10.2s
 ✔ Network hengshi_default      Removed    0.1s

6. Pause the service

docker-compose stop  # Stop container services

If the operation is successful, the following messages will appear.

docker-compose stop
[+] Running 8/8
 ✔ Container hengshi-monit      Stopped   0.0s
 ✔ Container hengshi-sense      Stopped   0.0s
 ✔ Container hengshi-apmserver  Stopped   0.0s
 ✔ Container hengshi-engine     Stopped  10.6s
 ✔ Container hengshi-minio      Stopped  10.5s
 ✔ Container hengshi-flink      Stopped  10.4s
 ✔ Container hengshi-metadb     Stopped  10.4s
 ✔ Container hengshi-redis      Stopped  10.5s

7. Restart paused services

docker-compose start # Restart paused containers

If the operation is successful, the following messages will appear.

docker-compose start
[+] Running 8/8
 ✔ Container hengshi-flink      Started  1.2s
 ✔ Container hengshi-metadb     Started  0.9s
 ✔ Container hengshi-apmserver  Started  1.0s
 ✔ Container hengshi-redis      Started  1.0s
 ✔ Container hengshi-engine     Started  0.9s
 ✔ Container hengshi-monit      Started  1.2s
 ✔ Container hengshi-minio      Started  0.8s
 ✔ Container hengshi-sense      Started  0.7s

Example: .env File

# hengshi-sense image tag version
HS_IMAGE_TAG=$TAG

# Optional, configure the root path of the volume, default is under the docker-compose.yaml directory. Configuration is optional. Please plan the data disk, it is recommended to configure the largest data disk.
HSHOME=.

# hengshi web service port
HS_HENGSHI_EXPOSE_PORT=8080

# hengshi data-gateway service port
DATA_GATEWAY_PUBLIC_EXPOSE_PORT=8079

# Optional, external exposed metadb port for the container, default is 54320
HS_PG_EXPOSE_PORT=54320

# Optional, engine type options: greenplum|doris
HS_ENGINE_TYPE=greenplum

# Optional, if engine type is greenplum, external exposed port for the engine
HS_ENGINE_EXPOSE_PORT=15432

# Optional, if engine type is doris, external exposed port for the engine
#HS_ENGINE_EXPOSE_PORT=9030

# Remote debug port
HS_JDWP_EXPOSE_PORT=5005

# Optional, MINIO service exposed port
HS_MINIO_EXPOSE_PORT=9891

# Optional, MINIO console external exposed port
HS_MINIO_CONSOLE_EXPOSE_PORT=9892

# Optional, redis service exposed port
HS_REDIS_EXPOSE_PORT=6379

# Optional, flink service exposed port
HS_FLINK_EXPOSE_PORT=8082

# Optional, apmserver service exposed port
HS_APMSERVER_EXPOSE_PORT=8200

# MINIO connection address | login user | login key
MINIO_SERVER_HOST=hengshi-minio
MINIO_ROOT_USER=hengshi
MINIO_ROOT_PASSWORD=hengshi202020

Operations After Starting the Service

When the HENGSHI SENSE service is running, it is necessary to regularly back up data to prevent data loss and promptly clean up unused logs to free up storage space.

1. Regular Data Backup

   It is recommended to back up the database metadb daily. The backup can be stored on local devices or remote devices. Regular backups are suggested during non-peak business hours, such as midnight, to avoid affecting user service usage. The following example is a command to back up data to a remote device daily at midnight. For detailed parameter explanations, please refer to Data Backup.

   0 0 * * * docker exec -it hengshi-metadb /opt/hengshi/bin/dbbackup.sh -m metadb -l /opt/hsdata/backup

2. Regular Log Cleanup

   During operation, HENGSHI SENSE generates runtime logs, which need to be cleaned up regularly to free up storage space. The following example shows commands for daily scheduled cleanup of rolling logs in the internal database.

   0 0 * * * docker exec -it hengshi-engine /opt/hengshi/bin/clean_engine.sh -t -r >& /<PATH_TO_HOST_LOG>/clean_engine.log
   0 23,12 * * * docker exec -it hengshi-engine /opt/hengshi/bin/clean_engine.sh -c -g -p
   */5 * * * * docker exec -it hengshi-engine /opt/hengshi/bin/clean_engine.sh -l

Tip

Do not expose the HENGSHI service ports directly to the public network. It is recommended to provide services externally through nginx + domain name to avoid potential attacks caused by component vulnerabilities.