1. Problem description During a certain Linux operation, the function scp command was used, and the following directory could not be completed using Tab. The error was reported: -bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory Related links: character map file `UTF-8’ not found 2. Cause handling 1) As shown […]
Tag: etl
GaussDB database SQL series-A brief discussion on SQL and ETL
Table of contents 1. Foreword 2. Overview of SQL and ETL 3. SQL example in ETL process (GaussDB) 1. Extract 2. Transform 3. Load 4. Attached is an introduction to DataArts Studio 5. Summary 1. Foreword In SQL language, ETL (Extract, Transform, and Load) is a process used to extract data from a source system […]
Linux obtains network status, routing and other information (Rtnetlink)
1. RTnetlink Rtnetlink allows reading and changing the kernel routing table. It is used for communication between the kernel and various subsystems (routing subsystems, IP addresses, link parameters, etc.). User space can communicate with the kernel through the NET_LINK_ROUTER socket, which is based on standard netlink messages. Note: The usage of netlink is mentioned in […]
C language under Linux uses netlink sockets to communicate with the kernel module
Introduction to netlink Netlink socket is a special inter-process communication (IPC) used to communicate between user processes and kernel processes. It is also the most commonly used interface for network applications to communicate with the kernel. In the Linux standard kernel, the system integrates many netlink instances by default, such as log reporting, routing system, […]
Building a RealTime Streaming ETL Pipeline on AWS using
Author: Zen and the Art of Computer Programming 1. Introduction 1.1 Summary Many current Internet companies have scenarios that require real-time processing of data to meet business needs, such as news websites, e-commerce platforms, etc. Data sources are diverse and distributed on different servers. How can we quickly and efficiently extract data from the data […]
Linux deployment ETL tool kettle
1 System environment jdk1.8 kettle8 centos7 2 Configuration process 2.1 Configure jdk Check if there is a java environment java -version If not, execute the following command, if yes, skip it mkdir /usr/local/java cd /usr/local/java tar -zxvf jdk-8u101-linux-x64.tar.gz vim /etc/profile export JAVA_HOME=/usr/local/java/jdk1.8.0_101 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH export JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin export PATH=$PATH:${JAVA_PATH} source /etc/profile java -version date […]
Building Streaming ETL for MySQL and Postgres based on Flink CDC
Official website: https://ververica.github.io/flink-cdc-connectors/release-2.3/content/Quick Start/mysql-postgres-tutorial-zh.html The official tutorial has some pitfalls. After testing it myself, I took notes. Server environment: VM virtual machine: CentOS7.9 docker version: Docker version 24.0.5, build ced0996 docker-compose version: 2.19 jdk 1.8 Virtual machine IP:192.168.122.131 Memory: 16G (must be greater than or equal to 16G) CPU: 4g Disk: >= 60G 1. docker […]
This cloud ETL tool works with Python to easily implement large data set analysis, with cases attached.
1. Pain points of Python processing large data sets Python is one of the best tools for data analysis. Pandas, numpy, matplotlib, etc. are all data analysis tools in the Python ecosystem. However, processing large data sets is a major pain point of Python, especially when you perform IO operations on your local computer. It […]
[HSPICE simulation] Input netlist file (2) circuit topology description
Circuit topology description 1. Component description statement (Elements) Resistor Capacitors Inductors Diode MOSFET 2. Model (.MODEL) 2.1 Basic concepts 2.2 Use of models .model description statement .model statement for MOSFET 3. Power and stimulus description statements 3.1 Independent sources DC source Transient sources trapezoidal pulse source PWL Source Pattern Source 4. Calling library files 4.1 […]
Improved dung beetle optimization–adopting Levi’s flight and triangle wandering strategies
Today I bring you an improved dung beetle optimization algorithm. There was a previous article that introduced the improved sine algorithm guided dung beetle optimization algorithm (MSADBO), but this article has already been published. The improvements brought to you by the author today are entirely personal improvements and should not be found on the Internet. […]