canal [kə'næl],译意为水道/管道/沟渠,是阿里巴巴开源的产品,主要用途是基于 MySQL 数据库增量日志解析,提供增量数据订阅和消费。

canal 1.1.14版本已经原生支持Kafka和RocketMQ的消息投递,并且引入cannal-admin工程,可以在WebUI上进行动态管理能力,支持配置集群、server、instance等模块。

准备工作

  • 公司基于k8s进行环境应用管理,需提前准备好K8S集群
  • 采用RocketMQ进行消息管理,需要安装RocketMQ服务
  • canal配置文件和file数据、日志需要持久存储,提前准备好nfs
K8S集群安装

准备canal持久存储需要的PVC,这边使用storageclass模式:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$ cat canal-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: canal-data
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: managed-nfs-storage

canal启动的时候需要配置文件canal.properties,直接存放对应PVC目录下,register ip是canal自己的地址canal.canal.svc.cluster.local,admin地址我们定义为canal-admin.canal.svc.cluster.local:8089,如下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# register ip
canal.register.ip = canal.canal.svc.cluster.local

# canal admin config
canal.admin.manager = canal-admin.canal.svc.cluster.local:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster =

canal-admin的配置文件如下,端口默认8089,数据库使用我们已经安装好的,canal验证使用admin,密码admin,和前面注册时保持一致:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
server:
  port: 8089
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8

spring.datasource:
  address: mysql.databases.svc.cluster.local:3306
  database: canal_manager
  username: root
  password: 23985111
  driver-class-name: com.mysql.jdbc.Driver
  url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false
  hikari:
    maximum-pool-size: 30
    minimum-idle: 1

canal:
  adminUser: admin
  adminPasswd: admin

使用官方脚本进行,初始化数据库:https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart

canal-admin在K8S内以deployment方式部署,如下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
$ cat canal-admin-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: canal-admin
  labels:
    app: canal-admin
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: canal-admin
    spec:
      imagePullSecrets:
        - name: ram-secret
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: apps.k8s.icjl/devops
                operator: Exists
      containers:
      - name: canal-admin
        image: canal/canal-admin:v1.1.4
        imagePullPolicy: IfNotPresent
        ports:
        - name: canal-admin
          containerPort: 8089
        volumeMounts:
        - name: canal-data
          mountPath: /home/admin/canal-admin/conf/application.yml
          subPath: application.yml
      volumes:
      - name: canal-data
        persistentVolumeClaim:
          claimName: canal-data
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: canal-admin
  name: canal-admin
spec:
  ports:
  - name: canal-admin
    port: 8089
    targetPort: 8089
  selector:
    app: canal-admin

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: canal-admin
  labels:
    name: canal-admin
    ingress.k8s.icjl/business: assigned
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  tls:
  - hosts:
    - canal-admin-qa.icjl.net
    secretName: icjl-net-secret
  rules:
  - host: canal-admin-qa.icjl.net
    http:
      paths:
      - path: /
        backend:
          serviceName: canal-admin
          servicePort: canal-admin

启动admin

1
kubectl apply -f canal-admin-deployment.yaml -n canal

准备canal-server的deployment文件:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: canal
  labels:
    app: canal
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: canal
    spec:
      imagePullSecrets:
        - name: ram-secret
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: apps.k8s.icjl/devops
                operator: Exists
      containers:
      - name: canal
        image: canal/canal-server:v1.1.4
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: canal-data
          mountPath: /home/admin/canal-server/conf/canal.properties
          subPath: canal.properties
        - name: canal-data
          mountPath: /home/admin/canal-server/ins-conf/
          subPath: ins-conf
        - name: canal-data
          mountPath: /home/admin/canal-server/logs
          subPath: logs
        - name: canal-data
          mountPath: /home/admin/canal-server/data
          subPath: data
        ports:
        - name: canal-admin
          containerPort: 11110
        - name: canal-port
          containerPort: 11111
      volumes:
      - name: canal-data
        persistentVolumeClaim:
          claimName: canal-data

启动cannal-server

1
kubectl apply -f canal-deployment.yaml -n canal

打开canal-admin,我们使用的是ingress暴露出来的,默认登录密码admin/123456 。

可以看到已经自动注册进来一个canal-server了,可以对配置进行修改

采用RockerMQ做为消息投递,选择Server管理-操作-配置

1
2
3
4
5
6
7
canal.serverMode = RocketMQ
canal.file.data.dir = /home/admin/canal-server/ins-conf/

canal.mq.servers = rocketmq-a.mq.svc.cluster.local:9876,rocketmq-b.mq.svc.cluster.local:9876

canal.mq.producerGroup = hiningmeng

新建instance载入模板,进行配置

1
2
3
4
5
6
7
canal.instance.master.address=10.40.0.100:3306
canal.instance.dbUsername=root
canal.instance.dbPassword=root

canal.instance.filter.regex=uic\\.uic_merchant
canal.mq.dynamicTopic=uic\\.uic_merchant

查看日志和MQ是否成功,enjoy …