Overview This article introduces the use of ELK (elasticsearch, logstash, kibana) + kafka to build a log system. The main demonstration is to use spring aop to collect logs, and then send the logs to logstash through kafka. Logstash then writes the logs to elasticsearch, so that elasticsearch has log data. Finally, kibana is used […]
Tag: kafka
SpringBoot integrates Kafka (2)
Foreword This article mainly talks about SpringBoot integration with Kafka. If there is anything that needs improvement, please point it out Above link: SpringBoot integrates Kafka (1) About the author: Hello everyone, I am Qing Jin Blog homepage: CSDN homepage to tell stories Quote of the day: Work harder and be better Table of Contents […]
ELK log analysis system based on Kafka
1. Basic concepts ELK is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. ?Logstash is a server-side data processing pipeline that can ingest data from multiple sources simultaneously, transform the data, and send the data to a repository such as Elasticsearch. Kibana allows users to […]
On the love-hate relationship with kafka
Previous article We used kafka in the project, but a series of things that happened later made me understand kafka more deeply. Some time ago, there was a problem with online kafka. Messages accumulated and were not consumed. Will consumption start after restarting the service? Kafka in the production environment stops consuming after a period […]
kafka: partitioning strategy for producers to send messages
In order to increase the scalability of the system, Kafka introduces the concept of partitioning. The partitioning mechanism in Kafka refers to dividing each topic into multiple partitions (Partitions), and each partition is an ordered set of message logs. Each message under a topic will only be saved in a certain partition, and will not […]
Docker builds kafka cluster and tests the full version
Build a complete version of kafka cluster with docker (windows) 1. Install docker desktop. Open the docker official website and download docker desktop. The URL is given directly here: Install Docker Desktop on Windows | Docker Docs As shown below, click to download. After downloading, click to run the exe file. We use the interactive […]
[Practical Combat-08] Flink consumes kafka custom serialization
Purpose Let the data consumed from kafka be directly converted into our objects mvn pom <!– Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, […]
CVE-2023-34040 Kafka deserialization RCE
Vulnerability description Spring Kafka is a module in the Spring Framework ecosystem that is used to simplify the process of integrating Apache Kafka in Spring applications. A record refers to a record in a Kafka message. In the affected version, ErrorHandlingDeserializer is not configured for records by default. When the user sets the container property […]
Building and using docker for windows kafka
Dockers Kafka setup First install docker for windows Start the cmd window, pull the Zookeeper and kafka images, and select the most downloaded image. docker search zookeeper docker search kafka We choose the first image Next, start pulling the zookeeper and kafka images. Here we pull wurstmeister/kafka. docker pull zookeeper docker pull wurstmeister/kafka Open the […]
SpringBoot multiple sets of Kafka configuration
SpringBoot multiple sets of Kafka configuration Single group Kafka configuration After many days, let’s take a dip. Scenario I need to monitor kafka messages for callback processing in my daily development process. However, different third-party services use different kafka clusters, so the default kafka configuration automatically read by Spring No, it only supports one group […]