Tip: After the article is written, the table of contents can be automatically generated. For how to generate it, please refer to the help document on the right.
Article directory
- 1. What is canal?
-
- Canal working principle
- 2. Usage steps
-
-
- 1. Use docker to create a database container
- 2. Turn on binlog mode
-
- 3. Canal container installation
- 4. Canal microservice construction
- Summarize
1. What is canal?
Canal can be used to monitor changes in database data to obtain new or modified data.
canal was proposed in response to Alibaba’s dual computer room deployment in Hangzhou and the United States, and the business needs for synchronization across computer rooms.
Alibaba companies began to gradually try database-based log analysis to obtain incremental changes for synchronization, which led to the incremental subscription & consumption business.
How Canal works
The principle is relatively simple:
- Canal simulates the interaction protocol of mysql slave, disguises itself as mysql slave, and sends the dump protocol to mysql master.
- Mysql master receives the dump request and starts pushing binary log to slave (that is, canal)
- canal parses the binary log object (originally a byte stream)
Canal needs to use mysql, but canal is implemented based on the master-slave mode of mysql, so binlog must be turned on first.
2. Usage steps
1. Use docker to create a database container
(1) Upload the mysql database configuration file mysqld.cnf
Go to the server’s /usr/mysql/conf directory (the directory mounted when creating the mysql container)
[mysqld] character-set-server=utf8 collation-server=utf8_general_ci pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = /var/lib/mysql symbolic-links=0
2. Turn on binlog mode
(1) Modify the mysqld.cnf configuration file and add the following configuration
The configuration above is as follows
log-bin=/var/lib/mysql/mysql-bin server-id=12345
Restart the mysql container and check whether the binlog is successfully opened.
(2) Use the root account to create a user and grant permissions (this step is very important, beginners must not forget it)
create user canal@'%' IDENTIFIED by 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT,SUPER ON *.* TO 'canal'@'%'; FLUSH PRIVILEGES;
(3) Restart the mysql container
3. canal container installation
Download image:
docker pull docker.io/canal/canal-server
Container installation
docker run –name canal -d -p 11111:11111 docker.io/canal/canal-server
Enter the container and modify the core configuration canal.properties and instance.properties. canal.properties is the configuration of canal itself, and instance.properties is the database connection configuration that needs to synchronize data.
The execution code is as follows:
docker exec -it canal /bin/bash
cd canal-server/conf/example
vi instance.properties
Modify instance.properties and configure the database connection address:
Note that the ip fills in the host ip bound to mysql: 192.168.1.1 (everyone is different)
After the configuration is completed, set it to boot and remember to restart canal.
docker update –restart=always canal
docker restart canal
Open cancal’s firewall port: (Ignore this step if the firewall is turned off)
firewall-cmd –add-port=11111/tcp –permanent
firewall-cmd –reload
If the connection to canal fails, check the log.
/home/admin/canal-server/logs/example
example.log
4. Canal microservice construction
(1) Install the auxiliary jar package (equivalent to the starter for springboot integrating canal)
(2) The canal microservice project builds the canal-service project and introduces relevant configurations.
pom.xml
<dependencies>
<!--web starting dependencies-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!--canal dependency-->
<dependency>
<groupId>com.xpand</groupId>
<artifactId>starter-canal</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<!--nacos service discovery-->
<dependency>
<groupId>com.alibaba.cloud</groupId>
<artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
</dependencies>
application.yml configuration
server:
port: 9003
spring:
application:
name:canal
cloud:
nacos:
discovery:
server-addr: localhost:8848
Feign:
hystrix:
enabled: true
#hystrix configuration
hystrix:
command:
default:
execution:
timeout:
#If enabled is set to false, the request timeout is handed over to the ribbon control
enabled: true
isolation:
strategy: SEMAPHORE
#canalconfiguration
Canal:
client:
instances:
example:
host: 192.168.1.1//Change to your own virtual machine port number
port: 11111
Note: Enable the port firewall of the canal server
firewall-cmd –add-port=11111/tcp –permanent
firewall-cmd –add-port=3306/tcp –permanent
firewall-cmd –reload
(3) Monitor creation
Create a CanalDataEventListener class to monitor table addition, deletion and modification operations. The code is as follows:
import com.alibaba.otter.canal.protocol.CanalEntry;
import com.xpand.starter.canal.annotation.*;
@CanalEventListener
public class CanalDataEventListener {<!-- -->
/***
* Add data monitoring
* @param eventType
* @param rowData
*/
@InsertListenPoint
public void onEventInsert(CanalEntry.EventType eventType, CanalEntry.RowData rowData) {<!-- -->
System.out.println(eventType.name());
for (CanalEntry.Column column : rowData.getAfterColumnsList()) {<!-- -->
System.out.println("Field name: " + column.getName() + " Field value: " + column.getValue());
}
}
/***
* Modify data monitoring
* @param rowData
*/
@UpdateListenPoint
public void onEventUpdate(CanalEntry.EventType eventType,CanalEntry.RowData rowData) {<!-- -->
System.out.println(eventType.name());
for (CanalEntry.Column column : rowData.getAfterColumnsList()) {<!-- -->
System.out.println("Field name: " + column.getName() + " Field value: " + column.getValue());
}
}
/***
* Delete data monitoring
* @param eventType
*/
@DeleteListenPoint
public void onEventDelete(CanalEntry.EventType eventType) {<!-- -->
System.out.println(eventType.name());
}
/***
The monitoring set by canal is to monitor all tables in all libraries. You can monitor specified libraries and specified tables through customized monitoring.
* Customized data modification monitoring
* @param eventType
* @param rowData
*/
//@ListenPoint(destination = "example", schema = "dongyimaindb", table = {"tb_content_category", "tb_content"}, eventType = CanalEntry.EventType.UPDATE)
public void onEventCustomUpdate(CanalEntry.EventType eventType, CanalEntry.RowData rowData) {<!-- -->
System.out.println(eventType.name());
for (CanalEntry.Column column : rowData.getAfterColumnsList()) {<!-- -->
System.out.println("Field name: " + column.getName() + " Field value: " + column.getValue());
}
}
}
(4)Start class creation
Create a startup class in com.offcn.canal with the following code:
@SpringBootApplication(exclude={<!-- -->DataSourceAutoConfiguration.class})
@EnableCanalClient //Enable the current application as a Canal client
public class CanalApplication {<!-- -->
public static void main(String[] args) {<!-- -->
SpringApplication.run(CanalApplication.class,args);
}
}
(5)Test
Start the canal microservice, and then modify the table data of any database. The background output of the canal microservice is as follows:
Summary
Tip: Here is a summary of the article:
For example: This article only briefly introduces the use of cancal. The specific configuration and operation are very different. It requires more practice and trying. If you have anything to add, you can also send a private message or comment. Come on, Ori will give it to you.