Introduction to Logstash Logstash is an open source, server-side data processing pipeline that can receive data from multiple sources, transform them, and finally send them to a specified type of destination. Logstash implements various functions through the plug-in mechanism. You can download plug-ins with various functions from Logstash Plugins · GitHub, or you can write […]
Tag: logstash
ELK (elasticsearch kibana Logstash) (script installation)
1. Introduction to ELK ELK is the collective name for three software, namely the abbreviation of three open source software: Elasticsearch, Logstash and Kibana. These three software are all open source software, usually used together, and have been successively owned by the company Elastic.co, so they are referred to as the ELK protocol stack. ELK […]
yum quickly installs zookeeper and Kafka cluster deployment es installation logstash installation kibina word segmenter redis
Zookeeper installation Kafka is based on Zookeeper to achieve distributed coordination, so you need to set up a Zookeeper node before building a Kafka node. Both Zookeeper and Kafka depend on JDK. I installed JDK first: Install jdk yum install java-1.8.0-openjdk* -y 1 [[email protected] ~]# java –version java 11.0.5 2019-10-15 LTS Java(TM) SE Runtime Environment […]
Install logstash to synchronize data and register system services in Windows environment
Install logstash to synchronize data and register system services in Windows environment This method is suitable for Windows environment. The same configuration file configures multiple pipes and configures system services to prevent the program from being killed. 1. Install logstash (1) Download the compressed package and modify the logstash-sample.conf file in the config folder after […]
A single logstash file collects multiple filebeat logs
One Background Description I have installed logstash now and configured only one file. I want to collect the logs of nginx and java at the same time. nginx should display the IP address, country and city, status code, etc., and java should display the specific content of the log. 2 nginx filebeat settings filebeat.inputs: # […]
Logstash configures the mysql data source. Mysql-connector-java-8.0.13.jar does not have read permission or does not load data.
Logstash configures mysql data source, mysql-connector-java-8.0.13.jar does not have read permission [2023-09-19T11:44:11,226][ERROR][logstash.javapipeline][main][63a9fa48345e7ff6aa5948ba8c28fc5cf05d96b9dd7e1d35dd28429202350ad3] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::Jdbc jdbc_user=>”root”, schedule=>”0 1 * * *”, jdbc_password=><password>, statement=>”SELECT * FROM *** “, jdbc_driver_library=>”mysql-connector-java-8.0.13.jar”, jdbc_connection_string=>”jdbc:mysql://127.0.0.1:3306/***?useUnicode=true &characterEncoding =UTF-8 & serverTimezone=UTC & useSSL=false & allowMultiQueries=true”, id=>”63a9fa48345e7ff6aa5948ba8c28fc5cf05d96b9dd7e1d35dd28429202350ad3″, jdbc_driver_class=>”com. mysql.cj.jdbc.Driver” , enable_metric=>true, codec=><LogStash::Codecs::Plain id=>”plain_5b2c39bc-2e29-46d5-a23c-26a60bc31ab8″, enable_metric=>true, […]
ElasticSearch7.x Series 3: Use of Logstash
Table of contents Preface Logstash configuration file Database driven Configuration file writing: Just take a look, the focus is on multi-table synchronization configuration Configuration file, read SQLserver database export to txt and ElasticSearch Input source explanation Output explanation start up Multi-table synchronization: important, very important Configuration explanation What does filter do? sql statement pit Database […]
Use of zookeeper and kafka, logstash reading and writing kafka (section 5)
highlight: a11y-dark theme: juejin 1: Introduction to kafka: Kafka is known as the next generation distributed messaging system. It is an open source project of the non-profit organization ASF (Apache Software Foundation, referred to as ASF). Open source software such as HTTP Server, Hadoop, ActiveMQ, and Tomcat all belong to the Apache Foundation. Open source […]
Logstash collects redis to implement multi-type data collection (section 4)
highlight: a11y-dark theme: juejin logstash collects logs and writes them to redis Deploy the redis service on one server, specifically for log caching. It is used in scenarios where the web server generates a large amount of logs. For example, the memory of the server below is about to be used up. The reason for […]
Logstash collects multiple types of logs, uses if judgment, converts Tomcat access logs to json and collects them (section 3)
highlight: a11y-dark theme: juejin 3. Collect logs through logstash 3.1. Collect single logs through logstash and use if judgment The prerequisite is that the logstash user has read permissions for the collected log files and write permissions for the written files. web1(106) Install openjdk apt install openjdk-8-jdk -y download Installation package: nginx, logstash cd /usr/local/src/ […]