Configuration File

The configuration file contains the configuration parameters for HENGSHI SENSE. You can modify the configuration file to change the service mode of HENGSHI SENSE.

Configuration File Description

The configuration files are located in the conf directory and include hengshi-sense-env.sh and engine-segment-hosts. The engine-segment-hosts file is used to store configuration information for the segment tables of the acceleration engine, while hengshi-sense-env.sh contains all configuration items except for the acceleration engine segment host list.

Tip

If conf/hengshi-sense-env.sh does not exist, you can generate this file by copying the conf/hengshi-sense-env.sh.sample template.
After modifying the configuration files, you need to restart the HENGSHI service for the changes to take effect.

Configuration for Acceleration Engine

The configuration for the acceleration engine is only required when installing the acceleration engine.

• Engine Management Module Master
  The engine management module master is a standalone machine. Its IP or hostname can be configured via the conf/hengshi-sense-env.sh configuration file using the HS_ENGINE_HOST parameter.

• Engine Data Query Module Segment
  The engine data query module segment can be configured on multiple machines. Its IP or hostname is configured via the conf/engine-segment-hosts configuration file, with each line representing the IP or hostname of a segment. By default, the system generates a configuration containing only localhost.

Data Directory Configuration Recommendations

Please refer to the following recommendations for data directory configuration.

• During initial installation, the program and system data require at least 3GB of space.
• If there are many uploaded files occupying significant space, it is recommended to configure the HS_HENGSHI_DATA directory on a large disk. System data, uploaded data, and acceleration engine data are all stored in this directory.
• If the acceleration engine needs to accelerate large datasets, it is recommended to configure the HS_ENGINE_HOME directory on a large disk. Once configured, the acceleration engine data will be stored independently of HS_HENGSHI_DATA in this directory.

Common Configuration Variables Description

All configuration items need to be written into the conf/hengshi-sense-env.sh file.

Parameter	Purpose
JAVA_HOME	The runtime Java environment directory.
HS_HENGSHI_PORT	HENGSHI WEB service port, default is 8080. BI prohibits root startup; for ports below 1024, please configure proxy forwarding, such as nginx.
HS_HENGSHI_DATA	The root directory for storing all HENGSHI data. By default, it is under the installation directory. If there is a large partition dedicated to storing data, this variable can be configured.
HS_PG_HOST	Internal database host. When not using HENGSHI metadb, this variable can be used to modify the configuration.
HS_PG_PORT	Internal database listening port, default is 54320. If it conflicts with existing service ports, this variable can be used to change the port.
HS_PG_DB	Internal database name.
HS_PG_USR	Internal database username.
HS_PG_PWD	Internal database password.
HS_ENGINE_HOME	Path to engine data files.
HS_ENGINE_HOST	Engine host. After configuration, the master module of the engine will be deployed on the corresponding machine. Default is localhost.
HS_ENGINE_SEGMENTS	Engine worker nodes, an array. Default is (localhost). For multiple nodes, an example would be (Node-A Node-B Node-C).
HS_ENGINE_PORT	Engine port.

The default maximum JVM memory for HENGSHI startup is 6G. If you need to modify it, use HENGSHI_JAVA_OPTS. For example, to change it to a minimum of 1G and a maximum of 4G, configure export HENGSHI_JAVA_OPTS="-Xms1g -Xmx4g" in conf/hengshi-sense-env.sh.

Password Encryption in Configuration Files

If there is a requirement to store only encrypted passwords in the conf/hengshi-sense-env.sh configuration file, you can follow the steps below:

1. Generate the encrypted password
   The following command will generate the encrypted password for 'abc123abc123abc123abc123abc123abc123abc123abc123':

cd /opt/hengshi;
bin/gete.sh "abc123abc123abc123abc123abc123abc123abc123abc123"
# Program output
U2FsdGVkX1/Ol74LahZWp6HlMspWmBHL5c4/s20Aw7o/z89q9pUITWfE/w2RCAu9
eyjEPtdGLOfXg3OGVcr9CJRPwgmpQWmWHvZ4j1aBmH8=

2. Replace the variable in the configuration file
   For example, replace the configuration for "HS_PG_RO_PWD":

HS_PG_RO_PWD="abc123abc123abc123abc123abc123abc123abc123abc123"
# Replace with
E_HS_PG_RO_PWD="U2FsdGVkX1/Ol74LahZWp6HlMspWmBHL5c4/s20Aw7o/z89q9pUITWfE/w2RCAu9
eyjEPtdGLOfXg3OGVcr9CJRPwgmpQWmWHvZ4j1aBmH8="

Note: If both plaintext and encrypted configurations exist, the encrypted configuration will take precedence. For example, if both HS_PG_RO_PWD and E_HS_PG_RO_PWD are configured, the final effective configuration will be "E_HS_PG_RO_PWD".

3. Supported Variable Mapping for Encrypted Configuration

Plaintext Variable	Encrypted Variable
HS_PG_PWD	E_HS_PG_PWD
HS_PG_RO_PWD	E_HS_PG_RO_PWD
HS_ENGINE_PWD	E_HS_ENGINE_PWD
HS_SYSLOG_PWD	E_HS_SYSLOG_PWD
HS_ACCESSLOG_PWD	E_HS_ACCESSLOG_PWD
QUARTZ_PWD	E_QUARTZ_PWD
GREENPLUM_QUERY_PWD	E_GREENPLUM_QUERY_PWD
GREENPLUM_ETL_PWD	E_GREENPLUM_ETL_PWD
MINIO_ROOT_PASSWORD	E_MINIO_ROOT_PASSWORD
REDIS_PASSWORD	E_REDIS_PASSWORD

Custom JDBC Configuration

HENGSHI SENSE data source comes with a default JDBC driver. If you need to use multiple versions of JDBC drivers simultaneously, you can follow the steps below:

1. Take hive1.1.1, 1.1.0, and 1.0.0 as examples. The system requires the default hive JDBC driver and these three versions of hive JDBC drivers to be used simultaneously. Extract the dependency packages for each version and place the jar files in a corresponding directory on the hengshi server:

• /opt/hengshi/jdbc-jar/hive1.1.1
• /opt/hengshi/jdbc-jar/hive1.1.0
• /opt/hengshi/jdbc-jar/hive1.0.0

2. Add a configuration file /opt/hengshi/jdbc-jar/jdbc_conf.json with the following content:

[
  {
    // Matches the options.type field of the corresponding data connection
    "name":"hive",
    "versions":[
      {
        // Specific version number
        "version":"1.1.1",
        // Path to the jar file
        "jdbcJarPath":"/opt/hengshi/jdbc-jar/hive1.1.1",
        // Fully qualified name of the driver class. If it matches the default version's class name, this can be omitted.
        "driverName":"org.apache.hive.jdbc.HiveDriver",
        "configs":{
          // hive1.1 does not support autoCommit settings, set to false here.
          // All additional HENGSHI-specific configurations must start with _hs_ to ensure they are removed before building the connection pool.
          "_hs_set_auto_commit":false
        }
      },
      // Multiple versions are placed in different elements of the versions array
      {
        "version":"1.1.0",
        "jdbcJarPath":"/opt/hengshi/jdbc-jar/hive1.1.0",
        "driverName":"org.apache.hive.jdbc.HiveDriver",
        "configs":{
          "_hs_set_auto_commit":false
        }
      },
      {
        "version":"1.0.0",
        "jdbcJarPath":"/opt/hengshi/jdbc-jar/hive1.0.0",
        "driverName":"org.apache.hive.jdbc.HiveDriver",
        "configs":{
          "_hs_set_auto_commit":false
        }
      }
    ]
  }
  // Multiple different connections are placed in different elements of the outermost array
  // , {...}
]

3. Add the configuration item export HENGSHI_JDBC_CONFIG=/opt/hengshi/jdbc-jar/jdbc_conf.json to the configuration file conf/hengshi-sense-env.sh, and restart the hengshi service.
4. When creating a new data connection, select the corresponding version. default refers to the internal version of hengshi, while others are custom driver versions.

Note

1. Cluster deployment: Perform the above operations on all [hengshi] nodes.

2. Docker container deployment: Modify the .env file in the deployment directory to configure HENGSHI_JDBC_CONFIG: /opt/hengshi/jdbc-jar/jdbc_conf.json. Restart the hengshi service after configuration.

3. Kubernetes cluster deployment: For the jar files and configuration file location, adjust the /opt/hengshi/jdbc-jar directory to /opt/hsdata/jdbc-jar. To add configuration items, modify the configmap.yaml configuration file to include HENGSHI_JDBC_CONFIG: /opt/hengshi/jdbc-jar/jdbc_conf.json. Restart the hengshi service after configuration.

Secondary Access Path Configuration

The requirement for secondary path access relies on reverse proxy tools such as Nginx, with different versions having different configuration methods.

Configuration Method One

This configuration method is applicable to version 5.0.2 and above.

Nginx Reference Configuration

The configuration process uses the /bi secondary path as an example.

• prefix style

   location /bi/ {             #<====[1/3] Path prefix
        proxy_set_header Accept-Encoding "";  #<====[2/3] Do not omit this; it is required when compression is enabled
        proxy_set_header X-HS-Sub-Path "/bi/"; #<====[3/3] /bi/ Path prefix, consistent with the location declaration
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/;
   }

• longest matching prefix style

   location ^~ /bi/ {             #<====[1/3] Path prefix
        proxy_set_header Accept-Encoding "";  #<====[2/3] Do not omit this; it is required when compression is enabled
        proxy_set_header X-HS-Sub-Path "/bi/"; #<====[3/3] /bi/ Path prefix, consistent with the location declaration
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/;
   }

• regex style

   location ~ ^/bi {             #<====[1/3] Path prefix
        proxy_set_header Accept-Encoding "";  #<====[2/3] Do not omit this; it is required when compression is enabled
        proxy_set_header X-HS-Sub-Path "/bi/"; #<====[3/3] /bi/ Path prefix, consistent with the location declaration
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        rewrite ^/bi/(.*) /$1 break;  #<==== Rewrite URI
        proxy_pass http://hengshi-server;  #<==== No URI at the end of the path; there is no trailing '/'
   }

Ingress Reference Configuration

During the configuration process, use the /bi secondary path as an example.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Accept-Encoding "";     #<====[2/3] Do not omit this; it is required when compression is enabled
      proxy_set_header X-HS-Sub-Path "/bi/";   #<====[3/3] /bi/ path prefix, consistent with the location declaration
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
spec:
  ingressClassName: nginx
  rules:
    - host: example.hengshi.com
      http:
        paths:
          - path: /bi/     #<====[1/3] Path prefix
            pathType: Prefix

Configuration Method Two

This configuration method is applicable to versions 5.0.1 and 5.0.

Nginx Reference Configuration

The configuration uses the /bi secondary path as an example.

   upstream hengshi-server {
         server 127.0.0.1:8080;
      }

• prefix style

   location /bi/ {             #<====[1/3] Path prefix, no regex matching
        proxy_set_header Accept-Encoding "";  #<====[2/3] Do not omit this; it is required when compression is enabled
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/;
        sub_filter_once on;
        sub_filter '<base href="/">' '<base href="/bi/" />';  #<====[3/3] /bi/ path prefix, consistent with the location declaration
    }

• regex style

    location ^/bi/(.*) {             #<====[1/4] Path rule
        proxy_set_header Accept-Encoding "";  #<====[2/4] Do not omit this; it is required when compression is enabled
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/$1;  #<====[3/4] Pass the remaining part of the path '$1' in the URI to the upstream
        sub_filter_once on;
        sub_filter '<base href="/">' '<base href="/bi/" />';  #<====[4/4] /bi/ path prefix, consistent with the location declaration
    }

• exact match (This type can only match one API address and cannot match other APIs; it is used for special scenarios)

    location = /bi/api/auth/login-info {       #<====[1/4] Exact match path
        proxy_set_header Accept-Encoding "";  #<====[2/4] Do not omit this; it is required when compression is enabled
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/api/auth/login-info;  #<====[3/4] Write the full path of the upstream
        sub_filter_once on;
        sub_filter '<base href="/">' '<base href="/bi/" />';  #<====[4/4] /bi/ path prefix, consistent with the location declaration
    }

Ingress Reference Configuration

Use the secondary path /bi as an example during configuration.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hengshi-sense
  namespace: hengshi
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Accept-Encoding "";
      proxy_hide_header Access-Control-Allow-Origin;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
      sub_filter_once on;
      sub_filter '<base href="/">' '<base href="/bi/" />';
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
    - host: example.hengshi.com
      http:
        paths:
          - path: /bi(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: hengshi-sense
                port:
                  number: 8080

Tip

The CONTEXT_PATH parameter is no longer effective in this version.