1. Download hadoop
https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/core/hadoop-3.3.6/
2. Configure environment variables
1. Configure java environment variables
2. Configure hadoop environment variables
export HADOOP_HOME=/usr/local/bigdata/hadoop-3.3.6 export HBASE_HOME=/usr/local/bigdata/hbase-2.5.6 export JAVA_HOME=/usr/local/jdk-11 export CLASSPATH=.:$JAVA_HOME/lib export PATH=.:$JAVA_HOME/bin:$HBASE_HOME/bin:$HADOOP_HOME/bin:$PATH
3. Configure host
192.168.42.142 node4
3. Modify the configuration file corresponding to hadoop
3.1. Create a directory under the hadoop directory
mkdir logs
mkdir data
mkdir -p data/namenode/
mkdir -p data/datanode
mkdir -p data/tmp
3.2. Modify hadoop-env.sh
Enter the etc/hadoop directory and modify the hadoop-env.sh file
export JAVA_HOME=/usr/local/jdk-11/ export HADOOP_HOME=/usr/local/bigdata/hadoop-3.3.6
3.3. Modify yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node4</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>/usr/local/bigdata/hadoop-3.3.6/etc/hadoop:/usr/local/bigdata/hadoop-3.3.6/share/hadoop/common/lib/*:/usr/local/bigdata/ hadoop-3.3.6/share/hadoop/common/*:/usr/local/bigdata/hadoop-3.3.6/share/hadoop/hdfs:/usr/local/bigdata/hadoop-3.3.6/share/hadoop/ hdfs/lib/*:/usr/local/bigdata/hadoop-3.3.6/share/hadoop/hdfs/*:/usr/local/bigdata/hadoop-3.3.6/share/hadoop/mapreduce/*:/usr /local/bigdata/hadoop-3.3.6/share/hadoop/yarn:/usr/local/bigdata/hadoop-3.3.6/share/hadoop/yarn/lib/*:/usr/local/bigdata/hadoop-3.3 .6/share/hadoop/yarn/*</value>
</property>
3.4. Modify hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>//usr/local/bigdata/hadoop-3.3.6/data/namenode</value> //Note that the previous part of the path is changed to your own
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>//usr/local/bigdata/hadoop-3.3.6/data/datanode</value> //Note that the previous part of the path is changed to your own
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.umaskmode</name>
<value>022</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
?<name>dfs.namenode.acls.enabled</name>
?<value>false</value>
</property>
<property>
<name>dfs.namenode.xattrs.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>http://node4:9870</value>
</property>
3.5. Modify mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
3.6. Modify core-site.xml
<property>
<name>hadoop.tmp.dir</name>
<value>//usr/local/bigdata/hadoop-3.3.6/data/tmp</value> //Note that the previous part of the path is changed to your own
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://node4:9000</value>
</property>
<property>
<name>hadoop.http.authentication.simple.anonymous.allowed</name>
<value>true</value>
</property>
<property>
<name>hadoop.proxyuser.hwf.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hwf.groups</name>
<value>*</value>
</property>
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
<description>The FileSystem for hdfs: uris.</description>
</property>
3.7. Add in the start-dfs.sh and stop-dfs.sh files
HDFS_DATANODE_USER=root HDFS_DATANODE_SECURE_USER=root HDFS_NAMENODE_USER=root HDFS_SECONDARYNAMENODE_USER=root
3.8. Add in start-yarn.sh and stop-yarn.sh
YARN_RESOURCEMANAGER_USER=root HADOOP_SECURE_DN_USER=root YARN_NODEMANAGER_USER=root
Note: This is a test environment, so use root directly. For production, you need to create corresponding users and user groups
4. Format files
Enter hdfs namenode -format format
hdfs namenode -format
2023-11-09 22:48:25,218 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215 2023-11-09 22:48:25,229 INFO util.GSet: Computing capacity for map INodeMap 2023-11-09 22:48:25,229 INFO util.GSet: VM type = 64-bit 2023-11-09 22:48:25,229 INFO util.GSet: 1.0% max memory 944 MB = 9.4 MB 2023-11-09 22:48:25,229 INFO util.GSet: capacity = 2^20 = 1048576 entries 2023-11-09 22:48:25,230 INFO namenode.FSDirectory: ACLs enabled? false 2023-11-09 22:48:25,230 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true 2023-11-09 22:48:25,230 INFO namenode.FSDirectory: XAttrs enabled? false 2023-11-09 22:48:25,231 INFO namenode.NameNode: Caching file names occurring more than 10 times 2023-11-09 22:48:25,235 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536 2023-11-09 22:48:25,236 INFO snapshot.SnapshotManager: SkipList is disabled 2023-11-09 22:48:25,239 INFO util.GSet: Computing capacity for map cachedBlocks 2023-11-09 22:48:25,239 INFO util.GSet: VM type = 64-bit 2023-11-09 22:48:25,239 INFO util.GSet: 0.25% max memory 944 MB = 2.4 MB 2023-11-09 22:48:25,239 INFO util.GSet: capacity = 2^18 = 262144 entries 2023-11-09 22:48:25,244 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 2023-11-09 22:48:25,244 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 2023-11-09 22:48:25,245 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 2023-11-09 22:48:25,247 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 2023-11-09 22:48:25,247 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 2023-11-09 22:48:25,248 INFO util.GSet: Computing capacity for map NameNodeRetryCache 2023-11-09 22:48:25,248 INFO util.GSet: VM type = 64-bit 2023-11-09 22:48:25,248 INFO util.GSet: 0.029999999329447746% max memory 944 MB = 290.0 KB 2023-11-09 22:48:25,248 INFO util.GSet: capacity = 2^15 = 32768 entries Re-format filesystem in Storage Directory root= /usr/local/bigdata/hadoop-3.3.6/data/namenode; location= null ? (Y or N) Y 2023-11-09 22:48:27,082 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1461429317-192.168.42.142-1699541307072 2023-11-09 22:48:27,082 INFO common.Storage: Will remove files: [/usr/local/bigdata/hadoop-3.3.6/data/namenode/current/VERSION, /usr/local/bigdata/hadoop- 3.3.6/data/namenode/current/seen_txid, /usr/local/bigdata/hadoop-3.3.6/data/namenode/current/fsimage_0000000000000000000.md5, /usr/local/bigdata/hadoop-3.3.6/data/ namenode/current/fsimage_0000000000000000000] 2023-11-09 22:48:27,093 INFO common.Storage: Storage directory /usr/local/bigdata/hadoop-3.3.6/data/namenode has been successfully formatted. 2023-11-09 22:48:27,129 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/bigdata/hadoop-3.3.6/data/namenode/current/fsimage.ckpt_000000000000000000 using no compression 2023-11-09 22:48:27,198 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/bigdata/hadoop-3.3.6/data/namenode/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds . 2023-11-09 22:48:27,206 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 2023-11-09 22:48:27,222 INFO namenode.FSNamesystem: Stopping services started for active state 2023-11-09 22:48:27,222 INFO namenode.FSNamesystem: Stopping services started for standby state 2023-11-09 22:48:27,228 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown. 2023-11-09 22:48:27,229 INFO namenode.NameNode: SHUTDOWN_MSG: /****************************************************** *********** SHUTDOWN_MSG: Shutting down NameNode at node4/192.168.42.142 *************************************************** **********/
5. Start the stand-alone version of hadoop
Enter the sbin directory and enter ./start-all.sh
[root@node4 sbin]# ./start-all.sh Starting namenodes on [node4] Last login: Thursday November 9 22:21:08 CST 2023pts/0 Starting datanodes Last login: Thursday November 9 22:22:49 CST 2023pts/0 Starting secondary namenodes [node4] Last login: Thursday November 9 22:22:51 CST 2023pts/0 Starting resource manager Last login: Thursday November 9 22:22:55 CST 2023pts/0 Starting node managers Last login: Thursday November 9 22:23:00 CST 2023pts/0
6. View the startup interface
http://192.168.42.142:8088/
http://192.168.42.142:9870/
The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Cloud native entry-level skills treeHomepageOverview 17,006 people are learning the system