Article directory Oracle CDC configuration (Non-CDB database) Step 1: Turn on archive logs Step 2: Create Tablespace tablespace Step 3: Create users and grant corresponding permissions Oracle CDC DataStream API implementation 1. Define OracleSource 2. Data processing 3. Sink to MySQL refer to Oracle CDC configuration (Non-CDB database) Step 1: Enable archive logs Log in […]
Tag: cdc
Flink CDC-MySQL CDC configuration and DataStream API implementation code…can monitor and collect multiple tables from multiple databases
Article directory MySQL CDC configuration Step 1: Enable binlog 1. Check whether MySQL binlog is enabled 2. If binlog is not enabled Step 2: Set the binlog format to row 1. Make sure MySQL’s binlog format is set to ROW 2. If not set to row Step 3: Create CDC user MySQL CDC DataStream API […]
[Big Data] CDC Technology: Change Data Capture
CDC Technology: Change Data Capture 1. What is CDC? 2. Batch processing vs CDC 3. Four CDC implementation methods 3.1 Table metadata 3.2 Table differences Table differences 3.3 Database trigger Trigger-based CDC 3.4 Database transaction log Log-based CDC 4.Oracle CDC detailed explanation 4.1 Oracle CDC mechanism 4.1.1 Synchronous Replication Synchronous Change Data Capture Configuration 4.1.2 […]
FlinkCDC reads PostgreSQL in real time
1. Preparation (PG version is 11.6) 1. Change the configuration file postgresql.conf [postgres@hostname data]$ vim postgresql.conf # Change wal log mode to logical wal_level = logical # minimal, replica, or logical # Change the maximum number of processes sent by wal (the default value is 10). This value is the same as the solts setting […]
Practical combat: Big data Flink CDC synchronizes Mysql data to ElasticSearch
Article directory Preface knowledge accumulation Introduction to CDC Types of CDC Comparison of common CDC plans Springboot connects to Flink CDC Environmental preparation Project construction Run locally Cluster operation Package the project and transfer the package to the cluster for startup Remotely deploy packages to the flink cluster write at the end Foreword In the […]
Building Streaming ETL for MySQL and Postgres based on Flink CDC
Official website: https://ververica.github.io/flink-cdc-connectors/release-2.3/content/Quick Start/mysql-postgres-tutorial-zh.html The official tutorial has some pitfalls. After testing it myself, I took notes. Server environment: VM virtual machine: CentOS7.9 docker version: Docker version 24.0.5, build ced0996 docker-compose version: 2.19 jdk 1.8 Virtual machine IP:192.168.122.131 Memory: 16G (must be greater than or equal to 16G) CPU: 4g Disk: >= 60G 1. docker […]
flink-cdc-oracle
flinkcdc extracts oracle data First, you need to set up the data in the database. The reference URL is as follows https://blog.csdn.net/weixin_46580067/article/details/124985447?ops_request_misc=%7B%22request%5Fid%22%3A%22169323165516800185874882%22%2C%22scm%22%3A%2220140713 .130102334.pc%5Fall.% 22%7D &request_id=169323165516800185874882 &biz_id=0 &utm_medium=distribute.pc_search_result.none-task-blog-2allfirst_rank_ecpm_v1~rank_v31_ecpm-2-124985447-null-nu ll .142v93insert_down28v1 &utm_term=flinkcdc extract oracle data code&spm=1018.2226.3001.4187 Directly upload the code package flinkcdc; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.TableResult; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; public class FlinkCdcOracle {<!– –> public static void main(String[] […]
In-depth interpretation of the design and implementation of MongoDB CDC
MongoDB CDC Overview MongoDB is a popular document-based non-relational database. MongoDB CDC [1] is a Flink connector provided by the Flink CDC community [2] for capturing change data (Change Data Capturing). It can be connected to the MongoDB database and Collect and capture the document addition, update, replacement, deletion and other change operations to generate […]
Use flink sqlserver cdc to synchronize data to StarRocks
Frontier: flink cdc is becoming more and more powerful and supports more data sources. This article introduces the implementation of flink cdc: sqlserver-》(using flink cdc)-》flink -》(using flink starrocks connector)->starrocks entire process 1. sqlserver environment preparation (you must use sqlserver versions below 16, flink cdc currently only supports sqlserver versions below 16) I am using the […]
flinkcdc data collection code FlinkAPI
1. flinkcdc data collection code: Background Use flinkcdc to collect mysql data to kafka. After two months of various debugging, I finally wrote the debugged version. I collected full and incremental data, and wrote a window every 10 minutes. Update the amount of data synchronized to each table once, implemented using FlinkAPI code Component version: […]