RocketMQ主要有四大部分:NameServer,Broker,Producer,Consumer。

RocketMQ的部署方式有多种:

  • 2m-noslave: 多Master模式
  • 2m-2s-sync: 多Master多Slave模式,同步双写
  • 2m-2s-async:多Master多Slave模式,异步复制

需要安装NameServer,Broker服务,本次安装采用2主模式,所以需要启动两个POD,每个Pod有两个container容器。

准备镜像

使用官方提供的rocketmqinc/rocketmq:latest,需要修改时区和yum源,方便后续容器调试。

From rocketmqinc/rocketmq

ENV TimeZone=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TimeZone /etc/localtime && echo $TimeZone > /etc/timezone

RUN rm /etc/yum.repos.d/* -rf

ADD CentOS-Base.repo  /etc/yum.repos.d/

RUN rm -rf rm -rf /opt/rocketmq-4.4.0/conf/2m* && \
    rm -rf /opt/rocketmq-4.4.0/conf/broker.conf && \
    rm -rf /opt/rocketmq-4.4.0/bin/*.cmd

#sudo docker build -t registry.cn-hangzhou.aliyuncs.com/hiningmeng/rocketmq:v4.4.0  .
准备配置文件

RocketMQ默认提供的配置文件都是最基本的,很多配置都是默认值,在生产环境中我们需要根据实际情况进行修改。样例配置如下:

$ cat broker-a.properties
#所属集群名字
brokerClusterName=mq-qa
#broker名字
brokerName=broker-a
#0表示Master,>0表示Slave
brokerId=0
#删除文件时间点,默认凌晨 4点
deleteWhen=04
#文件保留时间,默认 48 小时
fileReservedTime=48

brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
#nameServer地址,分号分割
namesrvAddr=rocketmq-a.mq.svc.cluster.local:9876;rocketmq-b.mq.svc.cluster.local:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
#commitLog每个文件的大小默认1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
mapedFileSizeConsumeQueue=300000

#-------------------

$ cat broker-b.properties
#所属集群名字
brokerClusterName=changjinglu-mq-qa
#broker名字
brokerName=broker-b
#0表示Master,>0表示Slave
brokerId=0
#删除文件时间点,默认凌晨 4点
deleteWhen=04
#文件保留时间,默认 48 小时
fileReservedTime=48

brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
#nameServer地址,分号分割
namesrvAddr=rocketmq-a.mq.svc.cluster.local:9876;rocketmq-b.mq.svc.cluster.local:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
#commitLog每个文件的大小默认1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
K8S安装RocketMQ

本次安装的服务使用独立的namespace。

kubectl create ns mq

broker的配置文件采用configmap的方式挂载到容器内,创建configmap的命令如下:

kubectl create configmap broker-config --from-file=broker-a.properties --from-file=broker-b.properties -n mq

启动命令-c可以指定配置文件,NameSvrAddr使用配置文件内指定的,配置文件使用前面的configmap挂载进来,subPath可以指定具体文件和目录,使用准备deployment文件如下:

$ cat rocketmq-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rocketmq-a
  namespace: mq
spec:
  replicas: 1
  template:
    metadata:
     labels:
       app: rocketmq-a
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: apps.k8s.icjl/devops
                operator: Exists
      imagePullSecrets:
        - name: ram-secret
      containers:
      - name: broker
        image: registry.cn-hangzhou.aliyuncs.com/hiningmeng/rocketmq:v4.4.0
        command: ["sh","-c","/opt/rocketmq-4.4.0/bin/mqbroker  -c /opt/rocketmq-4.4.0/conf/broker-a.properties"]
        imagePullPolicy: Always
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-brokeroptlogs
          - mountPath: /root/store
            name: rocketmq-data
            subPath: mq-brokeroptstore
          - name: broker-config
            mountPath: /opt/rocketmq-4.4.0/conf/broker-a.properties
            subPath: broker-a.properties
        env:
          - name: MAX_POSSIBLE_HEAP
            value: "4294967296"
      - name: namesrv
        image: registry.cn-hangzhou.aliyuncs.com/hiningmeng/rocketmq:v4.4.0
        command: ["sh","mqnamesrv"]
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 9876
        volumeMounts:
          - mountPath: /root/logs
            name: rocketmq-data
            subPath: mq-namesrvoptlogs
          - mountPath: /root/store
            name: rocketmq-data
            subPath: mq-namesrvoptstore
        env:
          - name: MAX_POSSIBLE_HEAP
            value: "1073741824"
      volumes:
      - name: rocketmq-data
        persistentVolumeClaim:
          claimName: rocketmq-data-a
      - name: broker-config
        configMap:
          name: broker-config
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmq-a
  name: rocketmq-a
  namespace: mq
spec:
  ports:
  - name: rocketmq-namesrv
    port: 9876
  selector:
    app: rocketmq-a

在deployment中使用到了persistentVolumeClaim,我采用的是storageclass的方式,具体使用可以自己学习下,需要准备pvc配置文件如下:

$ cat rocketmq-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: rocketmq-data-a
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: managed-nfs-storage

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: rocketmq-data-b
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: managed-nfs-storage
启动服务
kubectl apply -f rocketmq-pvc.yaml -n mq
kubectl apply -f rocketmq-master-a-deployment.yaml -n mq
kubectl apply -f rocketmq-master-b-deployment.yaml -n mq
测试消息发送
export NAMESRV_ADDR=rocketmq-a.mq.svc.cluster.local:9876
sh tools.sh org.apache.rocketmq.example.quickstart.Producer
sh tools.sh org.apache.rocketmq.example.quickstart.Consumer
安装console页面管理
$ cat rocketmq-admin-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rocketmq-admin
  namespace: mq
spec:
  replicas: 1
  template:
    metadata:
     labels:
       app: rocketmq-admin
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: apps.k8s.icjl/devops
                operator: Exists
      containers:
      - name: rocketmq-admin
        image: styletang/rocketmq-console-ng
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 8080
        env:
          - name: JAVA_OPTS
            value: "-Drocketmq.namesrv.addr=rocketmq-a.mq.svc.cluster.local:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmq-admin
  name: rocketmq-admin
  namespace: mq
spec:
  ports:
  - name: rocketmq-admin
    port: 8080
    targetPort: 8080
  selector:
    app: rocketmq-admin

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: mq
  name: rocketmq-admin
  labels:
    name: rocketmq-admin
    ingress.k8s.icjl/business: assigned
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  tls:
  - hosts:
    - rocketmq-admin.hiningmeng.cn
    secretName: icjl-net-secret
  rules:
  - host: rocketmq-admin.hiningmeng.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: rocketmq-admin
          servicePort: rocketmq-admin