1.1. 部署前准备

  1. 指定namespace, 比如hengshi
kubectl create namespace hengshi
  1. 根据版本获取docker-compose部署文件。
安装版本 部署文件 安装规模
3.x k8s 单机安装
4.0.x k8s 单机安装
4.1.x k8s 单机安装
3.x k8s 集群安装
4.0.x k8s 集群安装
4.1.x k8s 集群安装
  1. 替换gpdb.yaml中的$(POD_NAMESPACE) 变量为当前namespace, 比如 hengshi
sed -i 's/$(POD_NAMESPACE)/hengshi/'
  1. 修改pvc

  2. 以下服务需要修改volume:

    • 修改 storageClassName: csi-rbd, 为当前集群的storageclass
    • 修改 storage: , 为各个服务的存储大小
metadb.yaml
gpdb.yaml
zookeeper.yaml
redis.yaml
minio.yaml
flink.yaml

1.2. engine

1.2.1. 部署(engine)

  1. 初始化,并启动engine
kubectl -n hengshi apply -f gpdb.yaml
kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m initsystem
kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m startsystem
  • gpdb:6.2.1.1 及以前的版本, 在初始化执行以上命令后还要执行一下命令, 仅初始化执行一次即可.

    kubectl -n hengshi exec -it master-0 -- /bin/bash -c "source ~/.bashrc; /opt/hengshi/bin/engine.sh config"
    kubectl -n hengshi exec -it master-0 -- /bin/bash -c "source ~/.bashrc; psql -c \"ALTER USER \${GREENPLUM_USR} WITH SUPERUSER LOGIN PASSWORD '\${GREENPLUM_PWD}'\""
    
  • 如果需要修改 gpdb 的密码, 需要在两处修改:

    • gpdb.yaml
      GREENPLUM_PWD: hengshi202020
      GREENPLUM_QUERY_PWD: query202020
      GREENPLUM_ETL_PWD: etl202020
      
    • configmap.yaml
      HS_ENGINE_PWD: hengshi202020
      ENGINE_QUERY_PASSWORD: query202020
      ENGINE_ETL_PASSWORD: etl202020
      

1.3.1. nodePort

  1. 通过nodePort方式暴露flink的webUI
    #Client service for connecting to any flink Service.
    apiVersion: v1
    kind: Service
    metadata:
    name: flink
    spec:
    selector:
     hsapp: hengshi-sense
     hsrole: flink
    ports:
     - protocol: TCP
       name: "8082"
       port: 8082
       targetPort: 8082
       nodePort: 32584 # 集群外可访问的端口
    type: NodePort
    

1.3.2. ingress

  1. 通过ingress方式暴露flink的webUI

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
    name: hengshi-sense-flink
    namespace: hengshi-sense
    annotations:
     ingress.kubernetes.io/force-ssl-redirect: "false"
     nginx.ingress.kubernetes.io/proxy-connect-timeout: "90"
     nginx.ingress.kubernetes.io/proxy-send-timeout: "90"
     nginx.ingress.kubernetes.io/proxy-read-timeout: "90"
    spec:
    ingressClassName: nginx # 当前集群的ingressClass
    rules:
     - host: xxxx.hengshi.com # eg. flink.hengshi.com
       http:
         paths:
           - path: /
             pathType: Prefix
             backend:
               service:
                 name: flink
                 port:
                   number: 8082
    
  2. 修改configmap中的"FLINK_REST_WEB_URL" 变量为如下内容:

    FLINK_REST_WEB_URL: 'http://集群IP:nodePort端口' # 或是配置的ingress域名
    

1.4. hengshi

两种对外访问的示例配置,可根据需要选择其中之一即可

1.4.1. nodePort

  1. 通过nodePort方式暴露hengshi服务 ( 默认 )
    # Client service for connecting to any Hengshi Service.
    apiVersion: v1
    kind: Service
    metadata:
    name: hengshi
    spec:
    selector:
     hsapp: hengshi-sense
     hsrole: hengshi
    ports:    
    - protocol: TCP
     name: "8080"
     port: 8080
     targetPort: 8080
     #nodePort: 38080 # 指定集群外可访问的端口
    - protocol: TCP
     name: "5005"
     port: 5005
     targetPort: 5005
    - protocol: TCP
     name: "11111"
     port: 11111
     targetPort: 11111
    type: NodePort
    

1.4.2. ingress

  1. 通过ingress方式对外暴露hengshi服务 ( 可选 )
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
    name: hengshi-sense
    namespace: hengshi-sense
    annotations:
     ingress.kubernetes.io/force-ssl-redirect: "false"
     nginx.ingress.kubernetes.io/proxy-connect-timeout: "90"
     nginx.ingress.kubernetes.io/proxy-send-timeout: "90"
     nginx.ingress.kubernetes.io/proxy-read-timeout: "90"
    spec:
    ingressClassName: nginx # 当前集群的ingressClass
    rules:
     - host: xxxx.hengshi.com # eg. www.hengshi.com
       http:
         paths:
           - path: /
             pathType: Prefix
             backend:
               service:
                 name: hengshi-sense
                 port:
                   number: 8080
    
kubectl -n hengshi apply -f configmap.yaml
kubectl -n hengshi apply -f service.yaml
kubectl -n hengshi apply -f zookeeper.yaml
kubectl -n hengshi apply -f metadb.yaml
kubectl -n hengshi apply -f minio.yaml
kubectl -n hengshi apply -f redis.yaml
kubectl -n hengshi apply -f flink.yaml
kubectl -n hengshi apply -f hengshi.yaml
kubectl -n hengshi apply -f ingress.yaml #视情况是否部署ingress

1.5. 基本运维操作

1.5.1. 安全停止数据库服务(metadb,engine)

kubectl -n hengshi exec -it metadb-0 -- /docker-entrypoint.sh stop metadb single
kubectl -n hengshi exec -it master-0 -- /entrypoint.sh -m stopsystem

1.5.2. 重启(engine)

kubectl -n hengshi exec -it master-0 -- /entrypoint.sh gpstop -r

1.5.3. 扩容(engine)

  1. 修改 StatefulSet/segment
kubectl -n hengshi edit StatefulSet/segment
  • SEGMENTS字段填写扩容后所有segment 的appname (比如2个扩容到4个)
  • StatefulSet/segment 的 replicas: 改到扩容后所有segment数
apiVersion: v1
kind: ConfigMap
metadata:
  name: greenplum
data:
  MASTER: "master-0"
  SEGMENTS: |  #4个segment的列表
    segment-0
    segment-1
    segment-2
    segment-3
...
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: segment
spec:
  replicas: 4 #例如扩容后为4个segment
  • 然后 kubectl -n hengshi apply -f gpdb.yaml
  • 然后等待所有新增和旧有segement pod的状态都变成running

  • 写 new_host_file (新增segment列表, 比如原有2个segment(0,1), 现扩容到4个segment(0,1,2,3))

kubectl -n hengshi exec -it master-0 /bin/bash
cd /opt/hsdata/ && mkdir expand && cd expand
cat <<EOF > new_host_file
segment-2
segment-3
EOF
  1. 执行扩容操作
kubectl -n hengshi exec -it master-0 /bin/bash
cd /opt/hsdata/expand
psql postgres -c "create database expand"
gpexpand -f new_host_file  -D expand
  >y
  >0 #然后会生成 gpexpand_inputfile_yyyymmdd_xxxxxx 文件
gpexpand -i gpexpand_inputfile_yyyymmdd_xxxxxx -D expand
  1. 失败回滚(engine)
    kubectl -n hengshi exec -it master-0 /bin/bash
    cd /opt/hsdata/expand
    gpstart -aR
    gpexpand -r -D expand
    

1.5.4. engine数据迁移

  1. 旧engine数据导出
 # dump db data
kubectl exec -it $old-gp /bin/bash
source $HS_HOME/engine-cluster
pg_dumpall > /opt/hsdata/engine.back.sql
exit
  1. copy 数据到新机器
 # cp db data
kubectl cp $old-gp:/opt/hsdata/engine.back.sql engine.back.sql
kubectl cp engine.back.sql $master-0:/opt/hsdata/engine.back.sql
  1. 导入数据到新环境
 # load db data
kubectl exec -it $master-0 /bin/bash
source $HS_HOME/engine-cluster
psql postgres < /opt/hsdata/engine.back.sql
rm /opt/hsdata/engine.back.sql

1.6. 3.x 搭配 gpdb5 的配置 (hengshi 2.x 升级 3.x, 不升级gpdb)

  • 编辑 master-0 配置文件
kubectl -n hengshi exec -it metadb-0 -- /bin/bash
cd /opt/hengshi/conf; test -f hengshi-sense-env.sh || cp hengshi-sense-env.sh.sample hengshi-sense-env.sh
cat<<EOF >> hengshi-sense-env.sh
ENGINE_QUERY_USER=hengshi_query
ENGINE_QUERY_PASSWORD=query202020
ENGINE_QUERY_QUEUE=hengshi_query_queue
QUERY_QUEUE_ACTIVE_NUM=10
ENGINE_ETL_USER=hengshi_etl
ENGINE_ETL_PASSWORD=etl202020
ENGINE_ETL_QUEUE=hengshi_etl_queue
ETL_QUEUE_ACTIVE_NUM=4
EOF
  • 更新gpdb配置
kubectl -n hengshi get pod
kubectl cp hengshi-sense-xxxxxxxxx-xxxxx:/opt/hengshi/bin/engine.sh engine.31.sh
kubectl cp engine.31.sh master-0:/opt/hengshi/bin/engine.31.sh
kubectl -n hengshi exec -it metadb-0 -- /bin/bash
cd /opt/hengshi/bin
./engine.31.sh config
  • 更新configmap
set_kv_config() {
    local config_file="$1"
    local param="$2"
    local val="$3"
    # edit param=val if exist or insert new param=val
    grep -E "^\s*${param}\s*:" "${config_file}" > /dev/null \
                || sed -i "$ a ${param}: ${val}" "${config_file}"
}
set_kv_config configmap.yaml ENGINE_QUERY_USER hengshi_query
set_kv_config configmap.yaml ENGINE_QUERY_PASSWORD query202020
set_kv_config configmap.yaml ENGINE_QUERY_QUEUE hengshi_query_queue
set_kv_config configmap.yaml ENGINE_ETL_USER hengshi_etl
set_kv_config configmap.yaml ENGINE_ETL_PASSWORD etl202020
set_kv_config configmap.yaml ENGINE_ETL_QUEUE hengshi_etl_queue
kubectl -n hengshi apply -f configmap.yaml
  • 重启hengshi pod
kubectl rollout restart deployment/hengshi-sense

1.7. 部署成单机版 (POC)

1.7.1. 修改配置文件为单机配置

执行前确保 configmap.yaml, hengshi.yaml 等配置文件与 config_to_single.sh 在同一个目录下

./config_to_single.sh

1.7.2. 部署引擎

参考 引擎部署

1.7.3. 部署单机版服务

kubectl -n hengshi apply -f configmap.yaml
kubectl -n hengshi apply -f service.yaml
kubectl -n hengshi apply -f metadb.yaml
kubectl -n hengshi apply -f hengshi.yaml

results matching ""

    No results matching ""

    容器部署 helm部署(可选)