Configuration File

The configuration file stores the configuration parameters for HENGSHI SENSE. You can change the way HENGSHI SENSE operates by modifying the configuration file.

Configuration File Description

The configuration files are located in the conf path and include two files: hengshi-sense-env.sh and engine-segment-hosts. The engine-segment-hosts file is used to store the configuration information of the segment tables for the acceleration engine, while hengshi-sense-env.sh contains all configuration items except for the acceleration engine segment host list.

Tip

If conf/hengshi-sense-env.sh does not exist, you can generate this file by copying the conf/hengshi-sense-env.sh.sample template. After modifying the configuration files, you need to restart the HENGSHI service for the changes to take effect.

Configuration for Acceleration Engine

Configuration for the acceleration engine is only required when installing the acceleration engine.

• Engine Management Module Master The engine management module master is a standalone machine, and its IP or hostname can be configured through the HS_ENGINE_HOST in the conf/hengshi-sense-env.sh configuration file.
• Engine Data Query Module Segment The engine data query module segment can be configured on multiple machines, and their IPs or hostnames are configured through the conf/engine-segment-hosts configuration file, with each line representing the IP or hostname of a segment. By default, the system will generate a configuration containing only localhost.

Data Directory Configuration Recommendations

Please refer to the following recommendations for data directory configuration.

• During the initial installation, the program and system data require at least 3G of space.
• If there are many uploaded files that occupy a large amount of space, it is recommended to configure the HS_HENGSHI_DATA directory to a large disk. System data, uploaded data, and accelerated engine data are all stored in this directory.
• If you need the acceleration engine to accelerate large data, it is recommended to configure the HS_ENGINE_HOME directory to a large disk. After configuration, the engine acceleration data will be stored independently of HS_HENGSHI_DATA in this directory.

Common Configuration Variables Description

All configuration items need to be written into the conf/hengshi-sense-env.sh file.

Parameter	Purpose
JAVA_HOME	The runtime environment directory for Java.
HS_HENGSHI_PORT	HENGSHI WEB service port, default is 8080. BI prohibits root startup, for ports below 1024, please configure proxy forwarding, such as nginx, etc.
HS_HENGSHI_DATA	The root directory where HENGSHI stores all data, default is under the installation directory. When there is a large partition specifically for storing data, this variable can be configured.
HS_PG_HOST	Internal database host of the program. When not using HENGSHI metadb, the configuration can be modified through this variable.
HS_PG_PORT	Internal database listening port of the program, default is 54320. If it conflicts with existing service ports, the port can be changed through this variable.
HS_PG_DB	Database name of the internal database of the program.
HS_PG_USR	Username of the internal database of the program.
HS_PG_PWD	Password of the internal database of the program.
HS_ENGINE_HOME	Engine data file path.
HS_ENGINE_HOST	Engine host. After configuration, the master module of the engine will be deployed on the corresponding machine, default is localhost.
HS_ENGINE_SEGMENTS	Engine worker nodes, array, default is (localhost). Example for multiple nodes is (Node-A Node-B Node-C).
HS_ENGINE_PORT	Engine port.

The default maximum memory for HENGSHI startup JVM is 6G. If you need to modify it, please use HENGSHI_JAVA_OPTS. For example, to change it to a minimum of 1G and a maximum of 4G, configure export HENGSHI_JAVA_OPTS="-Xms1g -Xmx4g" in conf/hengshi-sense-env.sh.

Password Encryption in Configuration Files

If there is a need to store only encrypted passwords in the conf/hengshi-sense-env.sh configuration file, you can proceed as follows:

1. Generate the encrypted password The following will generate the encrypted password for 'abc123abc123abc123abc123abc123abc123abc123abc123'

cd /opt/hengshi;
bin/gete.sh "abc123abc123abc123abc123abc123abc123abc123abc123"
#Program output
U2FsdGVkX1/Ol74LahZWp6HlMspWmBHL5c4/s20Aw7o/z89q9pUITWfE/w2RCAu9
eyjEPtdGLOfXg3OGVcr9CJRPwgmpQWmWHvZ4j1aBmH8=

2. Replace the variable in the configuration file For example, replace the configuration of "HS_PG_RO_PWD"

HS_PG_RO_PWD="abc123abc123abc123abc123abc123abc123abc123abc123"
#Replace with
E_HS_PG_RO_PWD="U2FsdGVkX1/Ol74LahZWp6HlMspWmBHL5c4/s20Aw7o/z89q9pUITWfE/w2RCAu9
eyjEPtdGLOfXg3OGVcr9CJRPwgmpQWmWHvZ4j1aBmH8="

Note: If both plaintext/encrypted configurations exist, the encrypted configuration takes precedence. For example, if both HS_PG_RO_PWD and E_HS_PG_RO_PWD are configured, then "E_HS_PG_RO_PWD" will be effective.

3. Variable Mapping Supporting Encrypted Configuration

Plaintext Variable	Encrypted Variable
HS_PG_PWD	E_HS_PG_PWD
HS_PG_RO_PWD	E_HS_PG_RO_PWD
HS_ENGINE_PWD	E_HS_ENGINE_PWD
HS_SYSLOG_PWD	E_HS_SYSLOG_PWD
HS_ACCESSLOG_PWD	E_HS_ACCESSLOG_PWD
QUARTZ_PWD	E_QUARTZ_PWD
GREENPLUM_QUERY_PWD	E_GREENPLUM_QUERY_PWD
GREENPLUM_ETL_PWD	E_GREENPLUM_ETL_PWD
MINIO_ROOT_PASSWORD	E_MINIO_ROOT_PASSWORD
REDIS_PASSWORD	E_REDIS_PASSWORD

Custom JDBC Configuration

HENGSHI SENSE data source comes with a default JDBC driver. If you need to use more than two versions of the JDBC driver simultaneously, you can use the following method:

1. Taking hive1.1.1, hive1.1.0, and hive1.0.0 as examples, the system needs the default hive JDBC driver and these three versions of the hive JDBC driver to be used simultaneously. Unzip the dependency packages of the respective versions and place the jar files in a corresponding directory on the hengshi server, such as /opt/hengshi/jdbc-jar/hive1.1.1, /opt/hengshi/jdbc-jar/hive1.1.0, /opt/hengshi/jdbc-jar/hive1.0.0.
2. Add a configuration file /opt/hengshi/jdbc-jar/jdbc_conf.json with the following content:

[
  {
    // Same as the options.type field of the corresponding data connection
    "name":"hive",
    "versions":[
      {
        // Specific version number
        "version":"1.1.1",
        // Path to the jar package
        "jdbcJarPath":"/opt/hengshi/jdbc-jar/hive1.1.1",
        // Full name of the driver main class, can be omitted if it matches the default version class name.
        "driverName":"org.apache.hive.jdbc.HiveDriver",
        "configs":{
          // hive1.1 does not support autoCommit setting, set to false here.
          // All additional HENGSHI-specific configurations need to start with _hs_ to be removed before building the connection pool.
          "_hs_set_auto_commit":false
        }
      },
      // Multiple versions are placed under different elements of the versions array
      {
        "version":"1.1.0",
        "jdbcJarPath":"/opt/hengshi/jdbc-jar/hive1.1.0",
        "driverName":"org.apache.hive.jdbc.HiveDriver",
        "configs":{
          "_hs_set_auto_commit":false
        }
      },
      {
        "version":"1.0.0",
        "jdbcJarPath":"/opt/hengshi/jdbc-jar/hive1.0.0",
        "driverName":"org.apache.hive.jdbc.HiveDriver",
        "configs":{
          "_hs_set_auto_commit":false
        }
      }
    ]
  }
  // Multiple different connections are placed under different elements of the outermost array
  // , {...}
]

3. Add the configuration item export HENGSHI_JDBC_CONFIG=/opt/hengshi/jdbc-jar/jdbc_conf.json to the configuration file conf/hengshi-sense-env.sh and restart the hengshi service.
4. When creating a new data connection, select the corresponding version. Default refers to the internal version of hengshi, others are custom driver versions.

Note

1. Cluster deployment: Perform the above operations on all [hengshi] nodes.
2. Docker container deployment: For adding configuration items, modify the docker-compose.yaml configuration file, add HENGSHI_JDBC_CONFIG: /opt/hengshi/jdbc-jar/jdbc_conf.json under hengshi>environment, and restart the hengshi service.
3. k8s cluster deployment: For the location of jar packages and configuration files, adjust the above /opt/hengshi/jdbc-jar directory to /opt/hsdata/jdbc-jar. For adding configuration items, modify the configmap.yaml configuration file, add HENGSHI_JDBC_CONFIG: /opt/hengshi/jdbc-jar/jdbc_conf.json, and restart the hengshi service.

Flink Web Interface Open Access Configuration

# Default configuration is export FLINK_REST_BIND_ADDRESS=127.0.0.1
# Add the following configuration to conf/hengshi-sense-env.sh
export FLINK_REST_BIND_ADDRESS=0.0.0.0 # IP address 0.0.0.0 or the server's local IP

Note

The Flink Web Interface can be accessed without a password. It is not recommended to directly expose the Flink port to the public network, as this may increase the risk of malicious attacks on the server. If you need to use this feature, it is recommended to access it through a reverse proxy tool like nginx, adding configurations to avoid accessing the service directly through the port.

Secondary Access Path Configuration

The requirement for secondary path access relies on reverse proxy tools such as Nginx, and different versions have different configuration methods.

Configuration Method One

This configuration method is applicable to version 5.0.2 and above.

Nginx Reference Configuration

Using the /bi secondary path as an example during configuration.

• prefix style

   location /bi/ {             #<====[1/3] Path address prefix
        proxy_set_header Accept-Encoding "";  #<====[2/3] Do not omit, this configuration is needed when compression is enabled
        proxy_set_header X-HS-Sub-Path "/bi/"; #<====[3/3] /bi/ path address prefix, consistent with the location declaration
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/;
   }

• longest matching prefix style

   location ^~ /bi/ {             #<====[1/3] Path address prefix
        proxy_set_header Accept-Encoding "";  #<====[2/3] Do not omit, this configuration is needed when compression is enabled
        proxy_set_header X-HS-Sub-Path "/bi/"; #<====[3/3] /bi/ path address prefix, consistent with the location declaration
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/;
   }

• regex style

   location ~ ^/bi {             #<====[1/3] Path address prefix
        proxy_set_header Accept-Encoding "";  #<====[2/3] Do not omit, this configuration is needed when compression is enabled
        proxy_set_header X-HS-Sub-Path "/bi/"; #<====[3/3] /bi/ path address prefix, consistent with the location declaration
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        rewrite ^/bi/(.*) /$1 break;  #<==== Rewrite URI
        proxy_pass http://hengshi-server;  #<==== No URI at the end of the path, there is no trailing '/'
   }

Ingress Reference Configuration

The configuration uses the /bi secondary path as an example.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Accept-Encoding "";     #<====[2/3] Do not omit, this configuration is needed when compression is enabled
      proxy_set_header X-HS-Sub-Path "/bi/";   #<====[3/3] /bi/ path address prefix, consistent with the location declaration
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
spec:
  ingressClassName: nginx
  rules:
    - host: example.hengshi.com
      http:
        paths:
          - path: /bi/     #<====[1/3] Path address prefix
            pathType: Prefix

Configuration Method Two

This configuration method is applicable to versions 5.0.1 and 5.0.

Nginx Reference Configuration

The configuration uses the /bi secondary path as an example.

   upstream hengshi-server {
         server 127.0.0.1:8080;
      }

• prefix style

   location /bi/ {             #<====[1/3] Path address prefix, no regex match
        proxy_set_header Accept-Encoding "";  #<====[2/3] Do not omit, this configuration is needed when compression is enabled
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/;
        sub_filter_once on;
        sub_filter '<base href="/">' '<base href="/bi/" />';  #<====[3/3] /bi/ path address prefix, consistent with location declaration
    }

• regex style

    location ^/bi/(.*) {             #<====[1/4] Path address rule
        proxy_set_header Accept-Encoding "";  #<====[2/4] Do not omit, this configuration is needed when compression is enabled
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/$1;  #<====[3/4] Pass the part after the path in the uri '$1' to upstream
        sub_filter_once on;
        sub_filter '<base href="/">' '<base href="/bi/" />';  #<====[4/4] /bi/ path address prefix, consistent with location declaration
    }

• exact match (This type can only match one API address and cannot match other APIs, used for special scenarios)

    location = /bi/api/auth/login-info {       #<====[1/4] Exact matching path
        proxy_set_header Accept-Encoding "";  #<====[2/4] Do not omit, this configuration is needed when compression is enabled
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://hengshi-server/api/auth/login-info;  #<====[3/4] Write the full path of the upstream
        sub_filter_once on;
        sub_filter '<base href="/">' '<base href="/bi/" />';  #<====[4/4] /bi/ path address prefix, consistent with location declaration
    }

Ingress Reference Configuration

The configuration uses the /bi secondary path as an example.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hengshi-sense
  namespace: hengshi
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Accept-Encoding "";
      proxy_hide_header Access-Control-Allow-Origin;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
      sub_filter_once on;
      sub_filter '<base href="/">' '<base href="/bi/" />';
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
    - host: example.hengshi.com
      http:
        paths:
          - path: /bi(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: hengshi-sense
                port:
                  number: 8080

Tip

The CONTEXT_PATH parameter is no longer effective in this version.