HDFS Java API Programming

Level 1: File reading and writing Knowledge points 1.HDFS file creation and operation steps step1: Get the FileSystem object; step2: Write through FSDataOutputStream; step3: Output the file content through FSDataInputStream. Programming requirements Get the system settings of hadoop and create an HDFS file in it. The file path is /user/hadoop/myfile; Add the string https://www.educoder.net in […]

Flume construction and installation When uploading HDFS web pages, the connection was refused…Trying::1…telnet: connect to address::1: Connection refused

Table of Contents 1. Flume 1.Features of Flume: 2. What can Flume do? 3. Flume collection and storage 4. Flume’s three major components 5. Flume official website connection Chinese version 2. Install Flume (1) Upload and decompress the software package (2) Configure environment variables 3. Test Flume (1) Edit the Flume configuration file and start […]

Using Java API to operate HDFS

(1) Experimental principle The experimental principle of using Java API to operate HDFS is as follows: Configure the Hadoop environment: First, you need to configure the Hadoop environment, including setting the Hadoop installation path, configuring core-site.xml and hdfs-site.xml files so that Java programs can connect to HDFS. Introducing Hadoop dependencies: In Java projects, you need […]

Writing and using HDFS Java API

First Need to set up a Hadoop environment and start Hadoop If you haven’t built it yet, you can read this article: Hadoop cluster construction and configuration-CSDN Blog Here I use the idea under windows to connect the Hadoop of the virtual machine (1) Install hadoop under windows The installed Hadoop needs to be the […]

Java API to access HDFS

1. Download IDEA Download address: https://www.jetbrains.com/idea/download/?section=windows#section=windows Scroll below to use the free IC version. Run the downloaded exe file. Note that it is best not to install the installation path to the C drive. It can be changed to other disk, and other options can be checked as needed. 2. Create a Java project Run […]

ETL toolDatax-ETL-SqlServerToHDFS

Personal homepage–Personal homepage ? ? Thanks for the likes and attention, we will make a little progress every day! come on! ? Table of Contents Personal homepage–Personal homepage ? 1. Overview of DataX 1.1 Introduction to DataX 1.2 DataX framework 1.3 Functional limitations 1.4 Support Data Channels 2. Configuration example 2.1 Environmental information 2.2 SQLServer […]

Solving ImportError: HDFStore requires PyTables, “No module named tables“ problem importing

Table of Contents Solving ImportError: HDFStore requires PyTables, “No module named ‘tables’” problem importing Step 1: Install PyTables library Step 2: Confirm that the PyTables library has been installed correctly Step 3: Rerun the program summary Introduction to PyTables library Main features of PyTables PyTables installation Solve ImportError: HDFStore requires PyTables, “No module named ‘tables’” […]

HDFS cluster NameNode high availability transformation

Article directory background High availability transformation Program implementation Environmental preparation Configuration file modification Application configuration Cluster status verification High availability verification Background Assume that there are currently three ZooKeeper servers, namely zk-01/02/03, and several DataNode servers; Currently, the Namenode of the HDFS cluster does not have high availability configuration. The Namenode and Secondary Namenode are […]

springboot hdfs operation tool class

Import dependencies <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.7.3</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.7.3</version> </dependency> Download winutils.exe and configure environment variables GitHub – srccodes/hadoop-common-2.2.0-bin: hadoop-common-2.2.0/bin HADOOP_HOME: The directory where the file is located\hadoop-common-2.2.0-bin-master\bin Add hdfs configuration information hdfs: path: hdfs://localhost:9000 username: hadoop01 Write code package com.iproinfo.refinedanalysis.service; import org.apache.hadoop.fs.BlockLocation; import org.springframework.web.multipart.MultipartFile; import java.util.List; import java.util.Map; public interface HdfsService { […]