Hadoop cluster environment setup

cd ~/.ssh
ssh-keygen-trsa
cat ./id_rsa.pub >> ./authorized_keys

Next change the ssh configuration:

cd /etc/ssh
sudo vim sshd_config

Configuration changes are shown in the figure

Then change the configuration of ssh on the connecting host

sudo vim ssh_config

This must be changed! ! !
I’ve read a lot of blogs before, but the video tutorials didn’t teach them, so I have been unsuccessful in logging in after configuring password-free login – the default ssh public key is user@hostname, so I When connecting to moyao, I log in as tinfgxue@moyao, but my other machine is moyao@moyao, and the user tingxue does not exist at all, so it keeps failing. After being stuck for a long time, I suddenly realized it when I looked at the ssh_config configuration file – of course Host and User followed by your own hostname
Restart the ssh service after the configuration is complete

sudo service ssh restart

Configure ssh password-free login
Without configuring password-free operation, you need to enter the password of the corresponding host every time you connect to SSH, which is very troublesome.
Here is an example. Similar operations need to be done on mingxi, and the ssh secret key and public key must be generated on moyao and mingxi, and then the password-free operations must be configured on the other two machines respectively.

cd ~/.ssh
scp ~/.ssh/id_rsa.pub moyao:/home/moyao

This is the configuration on tingxue
Then go to moyao:

cd /home/moyao
cat id_rsa.pub >> ~/.ssh/authorized_keys

You can then choose to delete the id_rsa.pub file
Return to tingxue

ssh moyao

After the configuration is completed, the authorized_keys file of each host should contain the public keys of the three hosts

Hadoop configuration

The configuration file of hadoop is in ${HADOOP_HOME}/etc/hadoop. Before configuring Hadoop, you need to have hadoop permissions; secondly, the configuration must be changed to your own host name, do not copy it completely
This is the effect after we configured it on three hosts

As shown in the figure, if the user of the folder is root, use

sudo chown -R tingxue hadoop

Change the owner of the folder to tingxue – that is, an ordinary user


Among them, we need to modify the five files core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml, and workers

workers

This file needs to be configured with the host name of the hadoop cluster – provided it has been configured in /etc/hosts. In fact, this represents the host IP address.

vim workers

core-site.xml

vimcore-site.xml

Here is an explanation of the location of the configuration. Only the configuration code will be placed later.

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://tingxue:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<!--Configure the static user used for HDFS web page login to be atguigu-->
<property>
<name>hadoop.http.staticuser.user</name>
<value>tingxue</value>
</property>
</configuration>

hdfs-site.xml

vim hdfs-site.xml

<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>tingxue:9870</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>mingxi:9868</value>
</property>
</configuration>

yarn-site.xml

vim yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>moyao</value>
</property>
<!--Inheritance of environment variables-->
<property>
<name>yarn.nodemanager.env whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
<!-- Turn on the log aggregation function -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!-- Enable log aggregation server address -->
<property>
<name>yarn.log.server.url</name>
<value>http://tingxue:19888/jobhistory/logs</value>
</property>
<!-- Log directory -->
<property>
<name>yarn.log.server.url</name>
<value>/tmp/logs</value>
</property>
<!--Set the log retention time to 7 days -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>

</configuration>

mapred-site.xml

vim mapred-site.xml

<configuration>
      <property>
               <name>mapreduce.framework.name</name>
               <value>yarn</value>
       </property>
       <property>
               <name>mapreduce.jobhistory.address</name>
               <value>tingxue:10020</value>
       </property>
       <property>
               <name>mapreduce.jobhistory.webapp.address</name>
               <value>tingxue:19888</value>
       </property>
       <property>
               <name>yarn.app.mapreduce.am.env</name>
               <value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
       </property>
       <property>
               <name>mapreduce.map.env</name>
               <value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
       </property>
       <property>
               <name>mapreduce.reduce.env</name>
               <value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
       </property>
</configuration>

The configuration is done on one machine. After the configuration is completed, it needs to be distributed to the other two hosts – but don’t worry, write three scripts first, which will make things twice the result with half the effort!

Scripting

This script is what I learned while studying – from Dark Horse
After creation, be sure to make the script an executable file

sudo chmod +x filename

Just use this command, if not, just 777

xsync

#!/bin/bash
#1. Determine the number of parameters
if [ $# -lt 1 ]
then
echo Not Enough Arguement!
exit;
fi
#2. Traverse all machines in the cluster
for host in tingxue moyao mingxi
do
echo ==================== $host ====================
#3. Traverse all directories and send them one by one
for file in $@
do
#4. Determine whether the file exists
if [ -e $file ]
then
#5. Get the parent directory
pdir=$(cd -P $(dirname $file); pwd)
#6. Get the name of the current file
fname=$(basename $file)
ssh $host "mkdir-p $pdir"
rsync -av $pdir/$fname $host:$pdir
else
echo $file does not exists!
fi
done
done

This script synchronizes the selected files to the preset host through the rsync command

for host in tingxue moyao mingxi

This sentence is the host to be traversed–Remember to change it to your own
Place the script file in the /usr/bin directory. The usage method is command + the file name to be synchronized.
Note
Be sure to initialize the master node before starting Hadoop! ! !

cd /usr/local/hadoop
./bin/hdfs namenode -format

Start hadoop after initialization is complete – otherwise the NameNode will not start.
The problem may lie in creating files. The solution is to first run the file on the other two hosts. Create the hadoop folder in the /usr/local directory, and then authorize it to ordinary users – as mentioned above, then use the xsync command to synchronize

myhadoop.sh

#!/bin/bash
if [ $# -lt 1 ]
then
echo "No Args Input..."
exit ;
fi

case $1 in
"start")
echo " =================== Start hadoop cluster ==================="
echo " --------------- Start hdfs ---------------"
ssh tingxue "/usr/local/hadoop/sbin/start-dfs.sh"
echo " --------------- start yarn ---------------"
ssh moyao "/usr/local/hadoop/sbin/start-yarn.sh"
echo " --------------- Start historyserver ---------------"
ssh tingxue "/usr/local/hadoop/bin/mapred --daemon start historyserver"
;;
"stop")
echo " =================== Shut down the hadoop cluster ==================="
echo " --------------- Close historyserver ---------------"
ssh tingxue "/usr/local/hadoop/bin/mapred --daemon stop historyserver"
echo " --------------- close yarn ---------------"
ssh moyao "/usr/local/hadoop/sbin/stop-yarn.sh"
echo " --------------- close hdfs ---------------"
ssh tingxue "/usr/local/hadoop/sbin/stop-dfs.sh"
;;
*)
echo "Input Args Error..."
;;
esac

This script is the hadoop cluster startup script. If you do not use the script, according to the above configuration, you need to start hdfs now in tingxue, then start yarn in moyao, and then start the historical service in tingxue. It will be very troublesome. Use the script to solve it with one click.

# Start
myhadoop.sh start
# closure
myhadoop.sh stop

jpsall

#!/bin/bash
for host in tingxue moyao mingxi
do
echo =============== $host ===============
ssh $host /usr/lib/jvm/jdk8/bin/jps
done

After starting the service, you need to check the service startup status on each host. It is too troublesome to check one by one. Save it if you can.

At this point, the Hadoop environment has been set up. This blog may not have much innovation, and the level of detail may not be as detailed as some bloggers, but it is a personal summary of the construction process and a summary of the pitfalls I have encountered.