Use cdc technology to synchronize data in real time (canal) – monitor local data and synchronize data to the cloud database through MQ 1. Download the canal package and modify the configuration file conf -> instance.properties under example canal.instance.master.address=192.168.8.211:3306 Change the connection database path canal.instance.dbUsername=root user account to connect to the database canal.instance.dbPassword=123456 User password […]
Tag: cdc
Use of CDC in CView class (MFC)
Article directory 1.Preliminary knowledge 2. Experimental purpose 3.Experimental content 4. Code implementation 5. Running results 6. Summary 1. Difficulties encountered in the experiment How to set the color and width of the drawn curve How to draw axes How to graph a function 2. Experience 1. Preliminary knowledge MFC (Microsoft Foundation Classes) is a set […]
FlinkCDC for mysql to Clickhouse
Complete dependencies <dependencies> <!– https://mvnrepository.com/artifact/org.apache.flink/flink-core –> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> <version>1.13.0</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.12</artifactId> <version>1.13.0</version> </dependency> <!– <dependency>–> <!– <groupId>org.apache.flink</groupId>–> <!– <artifactId>flink-jdbc_2.12</artifactId>–> <!– <version>1.10.3</version>–> <!– </dependency>–> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-jdbc_2.12</artifactId> <version>1.13.0</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.13.0</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.12</artifactId> <version>1.13.0</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-api-java-bridge_2.12</artifactId> <version>1.13.0</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-common</artifactId> <version>1.13.0</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-planner_2.12</artifactId> <version>1.13.0</version> […]
kettle snapshot-based CDC
1. Input and output requirements for conversion ?The first step is to copy the data in the student_cdc table to the student_cdc_sanp1 table, making student_cdc_sanp1 the first snapshot of the student_cdc table, and at the same time output the data to the student_cdc_sync table. ?The second step is to insert, update, and delete the data […]
springboot integrates Flink-CDC
Article directory 1. Flink & Flink CDC official website 2. Introduction to CDC & Flink CDC 1. What is CDC 2. What is Flink CDC? 3. Supported connectors 3. Springboot integrates Filnk CDC 1. Official website example 2. Maven dependencies 1) Flink and Flink CDC version mapping 2) Specific maven dependencies 3) Project pitfalls 3. […]
[Time Zone] Flink JDBC and CDC time field time zone testing and time benchmark
Related articles: A brief analysis of the relationship between various time types and timezone 1. Test purpose and value 1. Test the time zone of the general database without time zone type. mysql timestamp(3) type postgres timestamp(3) type sqlserver datetime2(3) type oracle type TIMESTAMP(3) type In the following tests, they are all ts fields. 2. […]
[ACDC Data Set]: Preprocess ACDC cardiac 3D MRI image data set to VOC data set format, convert nii to jpg, label to png
[Segment Anything Model] Column link for segmentation, welcome to learn. [Blogger WeChat]cvxiaoyixiao This column is the introduction and preprocessing of public data sets and is being updated continuously. Article directory 1 ACDC data set introduction 2 ACDC data set sample 3 Preprocessing ACDC target 4 Processing result sample 5 Code 6 Divide test set and […]
Practical case: Sql client uses sql to operate FlinkCDC2Hudi and supports restoring hudi jobs from savepoint
Flink supports restoring jobs from savepoint in SQL Client starting from version 1.13. Introduction to flink-savepoint Next, we build an example of mysql cdc data entering the hudi data lake through kafka from Flink SQL Client. The overall process is as follows: In the second step above, we manually stopped the Flink task of kafka→hudi, […]
Teach you step by step how to use USB CDC+MSC composite device (based on stm32f407) – 2
Continued from the above: Teach you step by step how to use USB CDC + MSC composite device (based on stm32f407) This article mainly records some data captured during usb enumeration and some scsi protocols involved in usb msc. USB MSC Protocol Mainly include usb protocol, msc protocol, scsi protocol, and SDIO interface protocol. The […]
Flink CDC-SQL Server CDC configuration and DataStream API implementation code… can monitor and collect multiple tables of a database
Article directory SQL Server CDC configuration Step 1: Enable the CDC function of the specified database Step 2: Create a database role Step 3: Create file groups & files Step 4: Enable the CDC function of the specified table SQLServer CDC DataStream API implementation 1. Define SqlServerSource 2. Data processing 3. Sink to MySQL refer […]