Hadoop3.3.4 distributed installation

Installation prerequisites: The Java environment has been configured and password-free SSH login between all machines. Note: flinkv1, flinkv2, and flinkv3 below are aliases of the three servers. 1. Cluster deployment planning Note: NameNode and SecondaryNameNode should not be installed on the same server Note: ResourceManager also consumes a lot of memory. Do not configure it […]

Build a Hadoop fully distributed cluster

Table of Contents Preparations required for construction: Building process: 1. Install the virtual machine 2. Configure the network 3. Modify the host name 4. Bind the host name and IP 5. Configure password-free login 6. Use remote connection tools to upload jdk and hadoop 7. Install jdk and Hadoop 1. Unzip jdk and hadoop 2. […]

[scray_redis distributed crawling case] — 17k novel

Note: These are only my personal case notes! ! ! No other guidance 1. Requirement Description_Web Page Analysis_Preparation 1.1 Requirements Description Website: [Boys_Novel Classification_Complete Novel Classification_Free Novel Classification-17K Novel Network] Requirements: Open the homepage—->Select: Category—->Select: Completed, only view free Get the name, chapter name, and chapter url of the book This is temporarily called the […]

Hadoop 3.3.6 distributed cluster environment construction

1 System Configuration A total of 3 centOS virtual machines have been prepared, master, slave1, slave2 Configure hosts resolution vim /etc/hosts 192.168.10.11 master 192.168.10.12 slave1 192.168.10.13 slave2 Modify hostname #Execute the following commands on the corresponding server hostnamectl set-hostname master hostnamectl set-hostname slave1 hostnamectl set-hostname slave2 Turn off firewall #Close firewall systemctl stop firewalld.service #Disable […]

Design practice of tens of millions of high-availability distributed reconciliation systems

Programmer Hei Ge 2023-11-10 15:55 Published in Hunan included in collection #Code 14 #Design1 #Java218 #programming141 #programmer157 Background At present, online business volume is increasing day by day, with daily orders exceeding tens of millions, capital flows are large, and capital security has become a key concern. In order to ensure the accuracy of each […]

MyBatis Plus integrates Redis to implement distributed second-level cache

MyBatis cache description MyBatis provides two levels of cache, namely first-level cache and second-level cache. The first-level cache is a SqlSession-level cache. It only stores cached data inside the SqlSession object. If the SqlSession objects are different, the cache cannot be hit. The second-level cache is a mapper-level cache. As long as the Mapper class […]

WSL + Vscode one-stop to build Hadoop pseudo-distributed + Spark environment

Wsl + Vscode one-stop to build Hadoop + Spark environment If you want to build an environment such as Linux, Hadoop, Spark, etc., the common practice now is to install a virtual machine on VM, Virtualbox and other software This article introduces how to build a relevant environment on the windows subsystem (Windows Subsystem for […]