相信很多人对阿里开源的canal和apacheflume都不陌生。
flume-canal-source是对flume的source扩展。从canal获取数据到flumechannel。进而可以实现binlog数据到kafka/hdfs/hive/elasticsearch等等。
如何使用部署canal、flume这里忽略。
配置flume配置source类型*
agent.sources = canalSourceagent.sources.canalSource.type = com.weiboyi.etl.flume.source.canal.CanalSource配置连接canal的三种方式*
canalzookeeperservers
agent.sources.canalSource.zkServers = zookeeper-host:2181canalserverurls
agent.sources.canalSource.serverUrls = canal-server1:111111,canal-server2:111111canalserverurls
agent.sources.canalSource.serverUrl = canal-server1:111111配置canaldestination*
agent.sources.canalSource.destination = example配置用户名密码
agent.sources.canalSource.username = useragent.sources.canalSource.password = passwdbinlogbatchsize,default1024
agent.sources.canalSource.batchSize = 1024是否需要MySQL修改前的数据,defaulttrue
agent.sources.canalSource.oldDataRequired = true
评论