1.1. 部署前准备

1. 指定namespace, 比如hengshi

kubectl create namespace hengshi

1. 根据版本获取k8s yaml部署配置文件。

1. 替换gpdb.yaml中的$(POD_NAMESPACE) 变量为当前namespace, 比如 hengshi

sed -i 's/$(POD_NAMESPACE)/hengshi/'

1. 修改pvc

2. 以下服务需要修改volume:

   • 修改 storageClassName: csi-rbd, 为当前集群的storageclass
   • 修改 storage: , 为各个服务的存储大小

metadb.yaml
gpdb.yaml
zookeeper.yaml
redis.yaml
minio.yaml
flink.yaml

1.2. engine

1.2.1. 部署(engine)

1. 初始化,并启动engine

kubectl -n hengshi apply -f gpdb.yaml
kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m initsystem
kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m startsystem

• gpdb:6.2.1.1 及以前的版本, 在初始化执行以上命令后还要执行一下命令, 仅初始化执行一次即可.

  kubectl -n hengshi exec -it master-0 -- /bin/bash -c "source ~/.bashrc; /opt/hengshi/bin/engine.sh config"
  kubectl -n hengshi exec -it master-0 -- /bin/bash -c "source ~/.bashrc; psql -c \"ALTER USER \${GREENPLUM_USR} WITH SUPERUSER LOGIN PASSWORD '\${GREENPLUM_PWD}'\""
  

• 如果需要修改 gpdb 的密码, 需要在两处修改:

  • gpdb.yaml

    GREENPLUM_PWD: hengshi202020
    GREENPLUM_QUERY_PWD: query202020
    GREENPLUM_ETL_PWD: etl202020
    

  • configmap.yaml

    HS_ENGINE_PWD: hengshi202020
    ENGINE_QUERY_PASSWORD: query202020
    ENGINE_ETL_PASSWORD: etl202020
    

1.3. flink

1.3.1. nodePort

1. 通过nodePort方式暴露flink的webUI

   #Client service for connecting to any flink Service.
   apiVersion: v1
   kind: Service
   metadata:
   name: flink
   spec:
   selector:
    hsapp: hengshi-sense
    hsrole: flink
   ports:
    - protocol: TCP
      name: "8082"
      port: 8082
      targetPort: 8082
      nodePort: 32584 # 集群外可访问的端口
   type: NodePort
   

1.3.2. ingress

1. 通过ingress方式暴露flink的webUI

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
   name: hengshi-sense-flink
   namespace: hengshi-sense
   annotations:
    ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "90"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "90"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "90"
   spec:
   ingressClassName: nginx # 当前集群的ingressClass
   rules:
    - host: xxxx.hengshi.com # eg. flink.hengshi.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: flink
                port:
                  number: 8082
   

2. 修改configmap中的"FLINK_REST_WEB_URL" 变量为如下内容:

   FLINK_REST_WEB_URL: 'http://集群IP:nodePort端口' # 或是配置的ingress域名
   

1.4. hengshi

两种对外访问的示例配置，可根据需要选择其中之一即可

1.4.1. nodePort

1. 通过nodePort方式暴露hengshi服务 ( 默认 )

   # Client service for connecting to any Hengshi Service.
   apiVersion: v1
   kind: Service
   metadata:
   name: hengshi
   spec:
   selector:
    hsapp: hengshi-sense
    hsrole: hengshi
   ports:    
   - protocol: TCP
    name: "8080"
    port: 8080
    targetPort: 8080
    #nodePort: 38080 # 指定集群外可访问的端口
   - protocol: TCP
    name: "5005"
    port: 5005
    targetPort: 5005
   - protocol: TCP
    name: "11111"
    port: 11111
    targetPort: 11111
   type: NodePort
   

1.4.2. ingress

1. 通过ingress方式对外暴露hengshi服务 ( 可选 )

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
   name: hengshi-sense
   namespace: hengshi-sense
   annotations:
    ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "90"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "90"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "90"
   spec:
   ingressClassName: nginx # 当前集群的ingressClass
   rules:
    - host: xxxx.hengshi.com # eg. www.hengshi.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hengshi-sense
                port:
                  number: 8080
   

kubectl -n hengshi apply -f configmap.yaml
kubectl -n hengshi apply -f service.yaml
kubectl -n hengshi apply -f zookeeper.yaml
kubectl -n hengshi apply -f metadb.yaml
kubectl -n hengshi apply -f minio.yaml
kubectl -n hengshi apply -f redis.yaml
kubectl -n hengshi apply -f flink.yaml
kubectl -n hengshi apply -f hengshi.yaml
kubectl -n hengshi apply -f ingress.yaml #视情况是否部署ingress

1.5. 基本运维操作

1.5.1. 安全停止数据库服务(metadb,engine)

kubectl -n hengshi exec -it metadb-0 -- /docker-entrypoint.sh stop metadb single
kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m stopsystem

1.5.2. 重启(engine)

kubectl -n hengshi exec -it master-0 -- /entrypoint.sh gpstop -r

1.5.3. 扩容(engine)

1. 修改 StatefulSet/segment

kubectl -n hengshi edit StatefulSet/segment

• SEGMENTS字段填写扩容后所有segment 的appname (比如2个扩容到4个)
• StatefulSet/segment 的 replicas: 改到扩容后所有segment数

apiVersion: v1
kind: ConfigMap
metadata:
  name: greenplum
data:
  MASTER: "master-0"
  SEGMENTS: |  #4个segment的列表
    segment-0
    segment-1
    segment-2
    segment-3
...
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: segment
spec:
  replicas: 4 #例如扩容后为4个segment

• 然后 kubectl -n hengshi apply -f gpdb.yaml

• 然后等待所有新增和旧有segement pod的状态都变成running

• 写 new_host_file (新增segment列表, 比如原有2个segment(0,1), 现扩容到4个segment(0,1,2,3))

kubectl -n hengshi exec -it master-0 /bin/bash
cd /opt/hsdata/ && mkdir expand && cd expand
cat <<EOF > new_host_file
segment-2
segment-3
EOF

1. 执行扩容操作

kubectl -n hengshi exec -it master-0 /bin/bash
cd /opt/hsdata/expand
psql postgres -c "create database expand"
gpexpand -f new_host_file  -D expand
  >y
  >0 #然后会生成 gpexpand_inputfile_yyyymmdd_xxxxxx 文件
gpexpand -i gpexpand_inputfile_yyyymmdd_xxxxxx -D expand

1. 失败回滚(engine)

   kubectl -n hengshi exec -it master-0 /bin/bash
   cd /opt/hsdata/expand
   gpstart -aR
   gpexpand -r -D expand
   

1.5.4. engine数据迁移

1. 旧engine数据导出

 # dump db data
kubectl exec -it $old-gp /bin/bash
source $HS_HOME/engine-cluster
pg_dumpall > /opt/hsdata/engine.back.sql
exit

1. copy 数据到新机器

 # cp db data
kubectl cp $old-gp:/opt/hsdata/engine.back.sql engine.back.sql
kubectl cp engine.back.sql $master-0:/opt/hsdata/engine.back.sql

1. 导入数据到新环境

 # load db data
kubectl exec -it $master-0 /bin/bash
source $HS_HOME/engine-cluster
psql postgres < /opt/hsdata/engine.back.sql
rm /opt/hsdata/engine.back.sql

1.6. 3.x 搭配 gpdb5 的配置 (hengshi 2.x 升级 3.x, 不升级gpdb)

• 编辑 master-0 配置文件

kubectl -n hengshi exec -it metadb-0 -- /bin/bash
cd /opt/hengshi/conf; test -f hengshi-sense-env.sh || cp hengshi-sense-env.sh.sample hengshi-sense-env.sh
cat<<EOF >> hengshi-sense-env.sh
ENGINE_QUERY_USER=hengshi_query
ENGINE_QUERY_PASSWORD=query202020
ENGINE_QUERY_QUEUE=hengshi_query_queue
QUERY_QUEUE_ACTIVE_NUM=10
ENGINE_ETL_USER=hengshi_etl
ENGINE_ETL_PASSWORD=etl202020
ENGINE_ETL_QUEUE=hengshi_etl_queue
ETL_QUEUE_ACTIVE_NUM=4
EOF

• 更新gpdb配置

kubectl -n hengshi get pod
kubectl cp hengshi-sense-xxxxxxxxx-xxxxx:/opt/hengshi/bin/engine.sh engine.31.sh
kubectl cp engine.31.sh master-0:/opt/hengshi/bin/engine.31.sh
kubectl -n hengshi exec -it metadb-0 -- /bin/bash
cd /opt/hengshi/bin
./engine.31.sh config

• 更新configmap

set_kv_config() {
    local config_file="$1"
    local param="$2"
    local val="$3"
    # edit param=val if exist or insert new param=val
    grep -E "^\s*${param}\s*:" "${config_file}" > /dev/null \
                || sed -i "$ a ${param}: ${val}" "${config_file}"
}
set_kv_config configmap.yaml ENGINE_QUERY_USER hengshi_query
set_kv_config configmap.yaml ENGINE_QUERY_PASSWORD query202020
set_kv_config configmap.yaml ENGINE_QUERY_QUEUE hengshi_query_queue
set_kv_config configmap.yaml ENGINE_ETL_USER hengshi_etl
set_kv_config configmap.yaml ENGINE_ETL_PASSWORD etl202020
set_kv_config configmap.yaml ENGINE_ETL_QUEUE hengshi_etl_queue
kubectl -n hengshi apply -f configmap.yaml

• 重启hengshi pod

kubectl rollout restart deployment/hengshi-sense

1.7. 部署成单机版 (POC)

1.7.1. 修改配置文件为单机配置

执行前确保 configmap.yaml, hengshi.yaml 等配置文件与 config_to_single.sh 在同一个目录下

./config_to_single.sh

1.7.2. 部署引擎

参考 引擎部署

1.7.3. 部署单机版服务

kubectl -n hengshi apply -f configmap.yaml
kubectl -n hengshi apply -f service.yaml
kubectl -n hengshi apply -f metadb.yaml
kubectl -n hengshi apply -f minio.yaml
kubectl -n hengshi apply -f redis.yaml
kubectl -n hengshi apply -f flink.yaml
kubectl -n hengshi apply -f hengshi.yaml