Tag: flume

Flume construction and installation When uploading HDFS web pages, the connection was refused…Trying::1…telnet: connect to address::1: Connection refused

Table of Contents 1. Flume 1.Features of Flume: 2. What can Flume do? 3. Flume collection and storage 4. Flume’s three major components 5. Flume official website connection Chinese version 2. Install Flume (1) Upload and decompress the software package (2) Configure environment variables 3. Test Flume (1) Edit the Flume configuration file and start […]

Flume installation configuration

Table of Contents Foreword: 1. Correspondence between java and Flume versions 2. Download Flume (both methods will work) (1) The first method is to download Flume in window and transfer it to the Linux virtual machine. (2) The second method is to directly use the wget command to install to the specified folder in the […]

Flume custom Sink collects txt data to Greenplum

1 FlumeInstallation Deployment < /strong> 1.1 Installation address (1) Flume official website address: Welcome to Apache Flume – Apache Flume (2) Document viewing address: Flume 1.11.0 User Guide – Apache Flume (3) Download address: http://archive.apache.org/dist/flume/ 1.2 Installation and Deployment (1) Upload apache-flume-1.9.0-bin.tar.gz to the /app/local directory of IDC 53 (2) Unzip apache-flume-1.9.0-bin.tar.gz to the /app/local […]

Graduation project – distributed real-time log analysis and intrusion detection system based on Flume+spark+Flask

Distributed real-time log analysis and intrusion detection system based on Flume + spark + Flask Introduction LogVision is a log analysis solution that integrates web log aggregation, distribution, real-time analysis, intrusion detection, data storage and visualization. Apache Flume is used for aggregation, Apache Kafka is used for distribution, Spark Streaming is used for real-time processing, […]

Real-time comprehensive case based on Flume+Kafka+Hbase+Flink+FineBI (4) Real-time computing requirements and technical solutions

Article directory 16: Real-time computing requirements and technical solutions 17: Basic introduction to Flink 18: Code module construction 19: Province parsing tool test 20: Flink code interpretation 21: Flink real-time computing test 16: Real-time computing requirements and technical solutions Goal: Understand real-time computing requirements and technical solutions Path step1: real-time calculation requirements step2: technical solution […]

Real-time comprehensive case (3) offline analysis based on Flume+Kafka+Hbase+Flink+FineBI

Article directory 08: Offline analysis: Hbase table design and construction 09: Offline analysis: Kafka consumer construction 10: Offline analysis: Hbase connection construction 11: Offline analysis: Construction of Rowkey 12: Offline analysis: Put data column construction 13: Offline Analysis: Storage Run Test 14: Offline analysis: Hive correlation test 15: Offline analysis: Phoenix correlation test 08: Offline […]

Real-time comprehensive case (2) data source based on Flume+Kafka+Hbase+Flink+FineBI

Article directory 04: Data source 05: Technical architecture and technology selection 06: Flume review and installation 07: Development of Flume collection program 04: Data source Objective: Understand the format of data sources and generate simulation data Path step1: data format step2: data generation Implementation Data format Message time Sender’s nickname Sender’s account number Sender’s gender […]

Flume installation and deployment

Flume installation and deployment 1. Installation and deployment (1) Upload apache-flume-1.10.1-bin.tar.gz to the /opt/software directory of Linux (2) Unzip apache-flume-1.10.1-bin.tar.gz to the /opt/module/ directory [xxx@hadoop102 software]$ tar -zxf /opt/software/apache-flume-1.10.1-bin.tar.gz -C /opt/module/ (3) Modify the name of apache-flume-1.10.1-bin to flume [xxx@hadoop102 module]$ mv /opt/module/apache-flume-1.10.1-bin /opt/module/flume (4) Modify the log4j2.xml configuration file in the conf directory and […]