--- url: /v6.2/en-us/k8s.md --- # Kubernetes Deployment Guide ## Pre-deployment Preparation 1. Obtain the **k8s** deployment configuration files for your version. | Version | Deployment File | Component Dependencies | |---------|-----------------|------------------------| | 5.3.x | [k8s-yaml](https://download.hengshi.com/k8s-docker-yaml/k8s-5.3.zip) | metadb, engine, hengshi, minio, redis, apm-server | | 5.4.x | [k8s-yaml](https://download.hengshi.com/k8s-docker-yaml/k8s-5.3.zip) | metadb, engine, hengshi, minio, redis, apm-server | | 6.0.x | [k8s-yaml](https://download.hengshi.com/k8s-docker-yaml/k8s-6.0.zip) | metadb, engine, hengshi, minio, redis, apm-server | | 6.1.x | [k8s-yaml](https://download.hengshi.com/k8s-docker-yaml/k8s-6.1.zip) | metadb, engine, hengshi, minio, redis, otelecol | | 6.2.x | [k8s-yaml](https://download.hengshi.com/k8s-docker-yaml/k8s-6.2.zip) | metadb, engine, hengshi, minio, redis, otelecol; the package includes `hengshi-rbac.yaml`, which grants the `default` ServiceAccount `namespaces/get` permission | > \[!WARNING] 6.2+ Upgrade Note > When upgrading from 6.1 to 6.2 or later, replace the namespace placeholder in `hengshi-rbac.yaml` and apply the file. See [6.2+ k8s Deployment Notes](install-upgrade-notice.md#6.2+-k8s-Deployment-Notes). 2. Import offline images and update the image addresses. ```shell wget https://download.hengshi.com/releases/hengshi-sense-xxx.tar.gz docker load -i hengshi-sense-xxx.tar.gz ``` > \[!TIP] Tip > The gpdb image address differs and must be replaced separately, e.g. image: gpdb:x.x.x\ > Replace all other components with the imported offline image tag, e.g. image: hengshi-sense:5.0-20231103-dp-427c5f\ > In k8s/helm environments, push images to the cluster’s registry, such as registry, harbor, Alibaba Cloud, or Tencent Cloud. 3. Replace the $(POD_NAMESPACE) variable in gpdb.yaml with the current namespace; the example below uses “hengshi”. ```shell sed -i 's/$(POD_NAMESPACE)/hengshi/' ``` 4. Modify PVCs * Change storageClassName: xxx to your cluster’s storageclass * Change storage: xxx to the desired size for each service * For the doris engine, edit doris.yaml ```shell metadb.yaml gpdb.yaml redis.yaml minio.yaml ``` 5. Create the namespace, e.g. hengshi ```shell kubectl create namespace hengshi ``` 6. Replace the namespace configuration in hengshi-rbac ```shell # Replace '__HENGSHI_NAMESPACE__' with the actual namespace name; 'hengshi' is used here as an example sed -i 's/__HENGSHI_NAMESPACE__/hengshi/' hengshi-rbac.yaml ``` 7. If you need to import self-signed certs, first place the certificate files in a local `./certs/` directory, then create a Secret named `trust-certs`. This command packages all regular files under `./certs/` into the Secret. ```shell kubectl -n hengshi create secret generic trust-certs --from-file=./certs/ ``` * The `trust-certs` Secret is mounted into the `hengshi` container at `/opt/hengshi/conf/trust-certs`. * The `hengshi` container automatically imports the certs in that directory when it starts. ## engine ### Deploy Engine To change the gpdb password, update it in two places: * gpdb.yaml ```shell GREENPLUM_PWD: hengshi202020 GREENPLUM_QUERY_PWD: query202020 GREENPLUM_ETL_PWD: etl202020 ``` * configmap.yaml ```shell HS_ENGINE_PWD: hengshi202020 ENGINE_QUERY_PASSWORD: query202020 ENGINE_ETL_PASSWORD: etl202020 ``` Initialize and start the engine. ```shell kubectl -n hengshi apply -f gpdb.yaml kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m initsystem kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m startsystem ``` > \[!TIP] Tip > doris engine yaml: doris.yaml does not require initsystem or startsystem operations ### Deploy Remaining Components Refer to the following deployment manifest YAML files. ```shell kubectl -n hengshi apply -f configmap.yaml kubectl -n hengshi apply -f service.yaml kubectl -n hengshi apply -f metadb.yaml kubectl -n hengshi apply -f minio.yaml kubectl -n hengshi apply -f redis.yaml kubectl -n hengshi apply -f hengshi-rbac.yaml kubectl -n hengshi apply -f hengshi.yaml kubectl -n hengshi apply -f ingress.yaml # Deploy this step as needed ``` > \[!TIP] Tip > configmap.yaml is the configuration file for the HENGSHI service. > > service.yaml defines the services for internal cluster communication and external exposure. > > ingress.yaml can be deployed as required. ### Exposing HENGSHI Services HENGSHI provides sample configurations for external access; choose whichever suits your needs. #### nodePort Expose the HENGSHI service via nodePort (default; if no ingress is configured, the nodePort exposed by the service can be used to provide external access). For example, the 8080 in the sample below corresponds to the nodePort mapping on the cluster. ```shell apiVersion: v1 kind: Service metadata: name: hengshi spec: selector: hsapp: hengshi-sense hsrole: hengshi ports: - protocol: TCP name: "8080" port: 8080 targetPort: 8080 - protocol: TCP name: "5005" port: 5005 targetPort: 5005 type: NodePort ``` #### ingress Expose the HENGSHI service externally via ingress (optional). ```shell apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hengshi-sense namespace: hengshi-sense annotations: ingress.kubernetes.io/force-ssl-redirect: "false" nginx.ingress.kubernetes.io/proxy-connect-timeout: "90" nginx.ingress.kubernetes.io/proxy-send-timeout: "90" nginx.ingress.kubernetes.io/proxy-read-timeout: "90" spec: ingressClassName: nginx rules: - host: xxxx.hengshi.com http: paths: - path: / pathType: Prefix backend: service: name: hengshi-sense port: number: 8080 ``` > \[!TIP] Tip > ingressClassName: `` > > host: `` ## Basic Operations & Maintenance ### Safely Stop the Database Services Refer to the following commands to stop metadb and engine. ```shell kubectl -n hengshi exec -it metadb-0 -- /docker-entrypoint.sh stop metadb single ``` ```shell kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m stopsystem ``` ### Restart Engine Refer to the following command to restart the engine. ```shell kubectl -n hengshi exec -it master-0 -- /entrypoint.sh gpstop -r ``` ### Log Cleanup During operation, HENGSHI SENSE generates runtime logs that must be periodically cleaned up to reclaim storage space. Below is an example command for cleaning the rolling logs of the internal database. ```shell kubectl -n hengshi exec -it master-0 -- /bin/bash crontab -e # Add the following cron lines, then save and exit 0 0 * * * /opt/hengshi/bin/clean_engine.sh -t -r -c -g -p */5 * * * * /opt/hengshi/bin/clean_engine.sh -l ``` ### Scale-out Engine 1. Edit StatefulSet/segment ```shell kubectl -n hengshi edit StatefulSet/segment ``` * Fill the SEGMENTS field with the appnames of **all** segments after scaling (e.g. 2 → 4) * Change StatefulSet/segment replicas to the **total** segment count after scaling ```shell apiVersion: v1 kind: ConfigMap metadata: name: greenplum data: MASTER: "master-0" SEGMENTS: | # list of 4 segments segment-0 segment-1 segment-2 segment-3 ... apiVersion: apps/v1 kind: StatefulSet metadata: name: segment spec: replicas: 4 # e.g. 4 segments after scaling ``` * Then `kubectl -n hengshi apply -f gpdb.yaml` * Wait until **all** new and existing segment pods are in `Running` state 2. Create new_host_file (list of **new** segments; e.g. original 2 segments (0,1) → scale to 4 segments (0,1,2,3)) ```shell kubectl -n hengshi exec -it master-0 /bin/bash cd /opt/hsdata/ && mkdir expand && cd expand cat < new_host_file segment-2 segment-3 EOF ``` 3. Perform the scale-out ```shell kubectl -n hengshi exec -it master-0 /bin/bash cd /opt/hsdata/expand psql postgres -c "create database expand" gpexpand -f new_host_file -D expand >y >0 # generates gpexpand_inputfile_yyyymmdd_xxxxxx gpexpand -i gpexpand_inputfile_yyyymmdd_xxxxxx -D expand ``` If scaling fails, roll back the engine with: ```shell kubectl -n hengshi exec -it master-0 /bin/bash cd /opt/hsdata/expand gpstart -aR gpexpand -r -D expand ``` ### Engine Data Migration 1. Export data from the old engine ```shell # dump db data kubectl -n hengshi exec -it -- bash source $HS_HOME/engine-cluster pg_dumpall > /opt/hsdata/engine.back.sql exit ``` 2. Copy data to the new machine ```shell # cp db data # Copy the old engine data to local kubectl -n hengshi cp :/opt/hsdata/engine.back.sql engine.back.sql # Copy the data into the new engine environment kubectl -n hengshi cp engine.back.sql master-0:/opt/hsdata/engine.back.sql ``` 3. Import data into the new environment ```shell # load db data kubectl -n hengshi exec -it master-0 -- bash source $HS_HOME/engine-cluster psql postgres < /opt/hsdata/engine.back.sql rm /opt/hsdata/engine.back.sql ``` ## Deploy Single-Node Edition (POC) 1. Modify the configuration files for single-node setup Before running, ensure that configmap.yaml, hengshi.yaml, and other configuration files are in the same directory as config_to_single.sh. ```shell ./config_to_single.sh ``` 2. Deploy the engine Refer to [Engine Deployment](#engine) 3. Deploy the remaining components Refer to the following deployment manifest YAML files. ```shell kubectl -n hengshi apply -f configmap.yaml kubectl -n hengshi apply -f service.yaml kubectl -n hengshi apply -f metadb.yaml kubectl -n hengshi apply -f minio.yaml kubectl -n hengshi apply -f redis.yaml kubectl -n hengshi apply -f hengshi.yaml ```