Big Data Advanced – Hadoop Deployment

Big Data – Hadoop

Install the corresponding software
Related software installation
(1) Install VMware-workstation
Version: VM17
Address: Download VMware Workstation Pro | CN

After the installation is complete, open the software;
Help (H) / Enter License Key (E)
License: JU090-6039P-08409-8J0QH-2YR7F
(2) Install Xshell, Xftp
Official Website: Free for Home/School – NetSarang Website (xshell.com)
Version: xshell7, xftp7

Note: If the software prompts that it needs to be upgraded, then connect to the network; the whole process of the upgrade is automatic, and it can be used after the upgrade.

(2.1) Use Xshell7
Each session is an ssh connection to Linux, so to use xshell to connect to Linux, you need to create a session first.

illustrate:
?Name: the name of the session, you can customize it
?Protocol: default ssh
?Host: fill in the input box of ip, for example: 192.168.100.2 (the specific ip will be explained later)
?Other options: default

Click OK to successfully create the session

Basic information can be seen.

(2.2) Using Xftp7
The function of ftp is used for file transfer, which can upload local windows files to Linux

The operation used is consistent with Xshell, create a new session, fill in the name and host

Install Linux system
In the previously installed VMware software, install the Linux system.
Open the software, Ctrl + N (new virtual machine)

Many of the following and above are processed by default, and the places that need attention are marked with red arrows

At this point, the virtual machine is created, and the image file Centos7 is added to the virtual machine

Find the file CentOS-7-x86_64-DVD-2009.iso stored locally, and click OK.

Note: In the virtual machine, if you find that your cursor disappears, you can press Alt + Ctrl
Enter by default

After waiting for a period of time to install, come to the language selection page, slide the selection box on the left to the bottom, select Chinese – Simplified Chinese, and click Continue

After waiting for a while, come to [Installation Information Summary], mainly modify System > Installation Location | Network

First click I want to configure the partition, and then click Finish to come to the custom partition

The default partition habit can be:
~/boot: 500M
?swap: 2G
?/: All remaining disk space

Add in the order listed above

Click to start the installation

Next, set a password for root

The password here is 123456, click Finish, and then wait for the virtual machine to install.

After the installation is complete, there will be a restart option, click [Restart].

Wait for the page to load and log in

Enter username root Enter password 123456
After logging in, check the ip
ip addr

Find your own ip, then open xshell, connect to this virtual machine, and make sure that the virtual machine is turned on when connecting.

First change to static ip, then you need to configure the network card, ens33
vi /etc/sysconfig/network-scripts/ifcfg-ens33
A total of one modification, four additions at the end

The first three digits of ip can be viewed in the virtual machine through ip addr: 192.168.200

Explain what was modified above.
BOOTPROTO=”static”: Network mode, originally DHCP mode, now changed to static static mode
IPADDR: specified static ip
NETMASK: subnet mask, generally 255.255.255.0
GATEWAY: default gateway, the first three digits of ip + 2 [192.168.200] + [.2] -> 192.168.200.2
DNS1: DNS (Domain Name Server) 8.8.8.8 – Google’s domain name server or use 114.114.114.114 – a nationwide domain name server

Briefly introduce the use of vi command (using English input method):
?vi is a text editing tool that comes with centos, and the upgraded version is vim, which will be installed later
–vi is divided into two modes:
?Command line mode: use vi to edit files, the default is command line mode, press i to enter insert mode
?Insert mode: Insert mode to return to the command line mode and press Esc
– In command line mode:
?:w: save the file
?:wq: save and exit the file
?/[content]: Find the content in the file
?:set nu: display line number
?Move the cursor by pressing up, down, left, and right on the small keyboard

After modifying the content of the network card, :wq to save and exit, and then restart the network card
systemctl restart network
After restarting, the ip in xshell needs to be changed to the static ip we set: 192.168.200.80

connect again

Install frequently used tools below
yum -y install vim

Wait for the installation to complete before installing

yum -y install net-tools
After the installation is complete, there are two more commonly used commands vim and ifconfig
vim: advanced version of vi (the operation is the same)
ifconfig: View ip commands more concisely

Before deploying the environment, prepare the software and module folders
illustrate
All compressed packages are managed under the /opt/software directory
All installation paths are managed under the /opt/module directory
mkdir /opt/module

mkdir /opt/software
Connect xftp to the virtual machine too

The left side is the file in our windows system, and the right side is the file in Linux

Switch directory to /opt/software, move the required package into it, and drag and drop

Check that the file is indeed moved in

Modify the host name: master
hostnamectl set-hostname master

bash

Deploy jdk environment
jdk version: 8u311
decompress
tar -zxvf /opt/software/jdk-8u311-linux-x64.tar.gz -C /opt/module/
change file name
mv /opt/module/jdk1.8.0_311/ /opt/module/jdk8
Add jdk environment variable
vim /etc/profile
content added (end of file)
#jdk8

export JAVA_HOME=/opt/module/jdk8

export PATH=

PATH:

PATH:JAVA_HOME/bin

Refresh environment files
source /etc/profile
Check if it takes effect
java # Three arbitrary views are fine

javac

java-version

Deploy hadoop environment
Hadoop version: 2.10.1
decompress
tar -zxvf /opt/software/hadoop-2.10.1.tar.gz -C /opt/module/
change file name
mv /opt/module/hadoop-2.10.1/ /opt/module/hadoop2.10
Add hadoop environment variables
vim /etc/profile
content added (end)

hadoop2.10

export HADOOP_HOME=/opt/module/hadoop2.10

#jdk8

export JAVA_HOME=/opt/module/jdk8

export PATH=

PATH:

PATH:JAVA_HOME/bin:

HADOOP_HOME/bin:

HADOOPH?OME/bin:HADOOP_HOME/sbin

Refresh environment files
source /etc/profile
Check if the hadoop environment is in effect
hadoop version

Configure hadoop core files
The detailed role of each file reference: (148 messages) 1. The role of Hadoop main configuration files Hadoop five configuration files 1 Le’s Blog-CSDN Blog
path
cd /opt/module/hadoop2.10/etc/hadoop/
core-site.xml

<property>

<name>fs.defaultFS</name>

<value>hdfs://master:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/opt/module/hadoop2.10/data/tmp</value>

</property>

hadoop-env.sh export JAVA_HOME=/opt/module/jdk8 # about line 25 yarn-site.xml

<property>

<name>yarn.nodemanager.aux_services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.resourcemamager.hostname</name>

<value>master</value>

</property>

hdfs-site.xml

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

mapred-site.xml

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

slaves master

Add ssh password-free login
download ssh service
yum -y install openssh
Configure host mapping
vim /etc/hosts
content, add at the end
192.168.200.100 master
add ssh key
ssh-keygen -t rsa # Enter three times

Copy key

ssh-copy-id master

Enter yes

Enter password: 123456

Format namenode node
hadoop namenode -format
turn off firewall
systemctl stop firewalld # Turn off the firewall once

systemctl disable firewalld # Permanently disable the firewall

start hadoop
start-dfs.sh

start-yarn.sh
view process
jps
9751 SecondaryNameNode

9913 Resource Manager

10313 Jps

9450 NameNode

10011 NodeManager

9581 DataNodes
Web side: 192.168.200.100:50070
Port number: 50070

shutdown hadoop
stop-dfs.sh
stop-yarn.sh