Tag: hadoop

Hadoop fully distributed construction

Foreword There are always various problems encountered when configuring a Hadoop cluster. Here is a complete summary of the complete Hadoop fully distributed configuration process, hoping to help more people. Virtual machine preparation A distributed cluster requires three virtual machines. I won’t explain too much here, just clone them directly. 1. Modify the host name […]

Hadoop cluster environment setup

Hadoop environment setup write in front Statement and pitfall summary Software preparation Introduction to Hadoop Officially built Prerequisite environment preparation Install Ubuntu20.04 Server XShell remote connection Install jdk Install hadoop SSH Static IP configuration Cluster construction Cluster creation Modify hostname and IP address SSH configuration Hadoop configuration workers core-site.xml hdfs-site.xml yarn-site.xml mapred-site.xml Scripting xsync myhadoop.sh […]

Install stand-alone version of hadoop-3.3.6 on linux

1. Download hadoop https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/core/hadoop-3.3.6/ 2. Configure environment variables 1. Configure java environment variables 2. Configure hadoop environment variables export HADOOP_HOME=/usr/local/bigdata/hadoop-3.3.6 export HBASE_HOME=/usr/local/bigdata/hbase-2.5.6 export JAVA_HOME=/usr/local/jdk-11 export CLASSPATH=.:$JAVA_HOME/lib export PATH=.:$JAVA_HOME/bin:$HBASE_HOME/bin:$HADOOP_HOME/bin:$PATH 3. Configure host 192.168.42.142 node4 3. Modify the configuration file corresponding to hadoop 3.1. Create a directory under the hadoop directory mkdir logs mkdir data mkdir -p […]

Population mobility comprehensive data monitoring system based on Hadoop

Everyone can help like, collect, follow and comment Recommended subscription for wonderful columns: in the column below JAVA practical project Article directory JAVA practical projects 1. Project introduction 2. Development environment 3. Project display-Population flow comprehensive data monitoring system based on Hadoop 4. Code display 5. Project Summary Everyone can like, collect, follow, and leave […]

Hadoop fully distributed construction

Table of Contents Environmental preparation Install hadoop Usage example Note Unless otherwise specified, all operations in this article are performed on the root user of all nodes. The files needed in this article are placed in the /usr/local/software directory (this directory is created by yourself) Environment preparation First, the basic situation of my use of […]

HadoopApache Hadoop YARN

Personal homepage–Driving a tractor home_Linux, Java basic learning, big data operation and maintenance-CSDN blog ? Thanks for the likes and attention, we will make a little progress every day! come on! Table of Contents 1. Overview of YARN 2. YARN infrastructure 2.1 ResourceManager (RM) 2.1.1 Scheduler 2.1.2 ApplicationManager 2.2 ApplicationMaster (AM) 2.3 NodeManager (NM) 2.4 […]

Sports recommendation system based on Hadoop

Collect and follow to avoid getting lost, end of source code article Article directory 1. Project introduction 2. Development environment 3. Function introduction 2.3 Use case analysis 4. Core code 5. Effect drawing 6. Article Table of Contents 1. Project introduction With the improvement of living standards, people’s health awareness is getting stronger and stronger. […]

hadoop mapreduce api calls WordCount native and cluster code

Run code natively package com.example.hadoop.api.mr; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import java.io.IOException; public class WordCount { /** * Text: refers to StringWritable * (LongWritable, Text) Input on the map side: these two parameters will never change, Text: text data, LongWritable: offset (offset […]

Network detection data statistical analysis system based on hadoop

Recommended subscription for wonderful columns: on the homepage below Author’s homepage: Computer graduate student Brother Article directory 1. Project introduction 2. Development environment 3. System display – Hadoop-based network detection data statistical analysis system 4. Code display 5. Project Summary You can help like, collect, follow and comment 1. Project introduction 1. Research background With […]