Installation prerequisites: The Java environment has been configured and password-free SSH login between all machines. Note: flinkv1, flinkv2, and flinkv3 below are aliases of the three servers. 1. Cluster deployment planning Note: NameNode and SecondaryNameNode should not be installed on the same server Note: ResourceManager also consumes a lot of memory. Do not configure it […]
Tag: distribute
Build a Hadoop fully distributed cluster
Table of Contents Preparations required for construction: Building process: 1. Install the virtual machine 2. Configure the network 3. Modify the host name 4. Bind the host name and IP 5. Configure password-free login 6. Use remote connection tools to upload jdk and hadoop 7. Install jdk and Hadoop 1. Unzip jdk and hadoop 2. […]
SpringBoot distributed verification code login solution
Foreword In order to prevent the verification system from being violently cracked, many systems have added verification code verification. The more common one is the picture QR code. The safer one in the industry is the SMS verification code. Of course, there are also some puzzle verification codes and QR codes with artificial intelligence. Wait, […]
What are the problems with Redis distributed locking and how to solve them?
Suppose there is a scenario where you buy a ticket on a ticket purchasing software, but there are only one or a few tickets left. At this time, dozens of people are using this software to purchase tickets at the same time. Without considering any impact, the normal logic is to first determine whether there […]
[scray_redis distributed crawling case] — 17k novel
Note: These are only my personal case notes! ! ! No other guidance 1. Requirement Description_Web Page Analysis_Preparation 1.1 Requirements Description Website: [Boys_Novel Classification_Complete Novel Classification_Free Novel Classification-17K Novel Network] Requirements: Open the homepage—->Select: Category—->Select: Completed, only view free Get the name, chapter name, and chapter url of the book This is temporarily called the […]
Spring transactions and distributed transactions
1. Specific definition of transactions Transactions provide a mechanism to incorporate all operations involved in an activity into an indivisible execution unit. All operations that make up a transaction can only be submitted if all operations can be executed normally. As long as any one of the operations fails to execute (an exception occurs) ), […]
Hadoop 3.3.6 distributed cluster environment construction
1 System Configuration A total of 3 centOS virtual machines have been prepared, master, slave1, slave2 Configure hosts resolution vim /etc/hosts 192.168.10.11 master 192.168.10.12 slave1 192.168.10.13 slave2 Modify hostname #Execute the following commands on the corresponding server hostnamectl set-hostname master hostnamectl set-hostname slave1 hostnamectl set-hostname slave2 Turn off firewall #Close firewall systemctl stop firewalld.service #Disable […]
Design practice of tens of millions of high-availability distributed reconciliation systems
Programmer Hei Ge 2023-11-10 15:55 Published in Hunan included in collection #Code 14 #Design1 #Java218 #programming141 #programmer157 Background At present, online business volume is increasing day by day, with daily orders exceeding tens of millions, capital flows are large, and capital security has become a key concern. In order to ensure the accuracy of each […]
MyBatis Plus integrates Redis to implement distributed second-level cache
MyBatis cache description MyBatis provides two levels of cache, namely first-level cache and second-level cache. The first-level cache is a SqlSession-level cache. It only stores cached data inside the SqlSession object. If the SqlSession objects are different, the cache cannot be hit. The second-level cache is a mapper-level cache. As long as the Mapper class […]
WSL + Vscode one-stop to build Hadoop pseudo-distributed + Spark environment
Wsl + Vscode one-stop to build Hadoop + Spark environment If you want to build an environment such as Linux, Hadoop, Spark, etc., the common practice now is to install a virtual machine on VM, Virtualbox and other software This article introduces how to build a relevant environment on the windows subsystem (Windows Subsystem for […]