Java API to access HDFS

1. Download IDEA

Download address: https://www.jetbrains.com/idea/download/?section=windows#section=windows
Scroll below to use the free IC version.
Run the downloaded exe file. Note that it is best not to install the installation path to the C drive. It can be changed to other disk, and other options can be checked as needed.

2. Create a Java project

Run IDEA and create a new project.
Choose to create a maven project
In order to facilitate management, Hadoop is used as the parent project, so the project type of Hadoop is modified to pom. Find pom.xml and add the packaging tag and dependency management. The submodule only needs to add the dependency name and does not need to import the dependency version.

<packaging>pom</packaging>

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>3.2.2</version>
            </dependency>
            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>4.13.2</version>
                <scope>test</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

3. Create a new submodule

Right click on the hadoop file-new-module

View dependencies

Add Java class

4. HDFS operation example

1. Display all directories under the HDFS specified directory.
Add the required dependencies in pom.xml

<dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-configuration2</artifactId>
            <version>2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>3.3.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>3.3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.3.1</version>
        </dependency>

Reference Code:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.IOException;

public class Demon {<!-- -->
    public static void main(String[] args) throws IOException {<!-- -->
        System.setProperty("HADOOP_USER_NAME","root");
        //Access hadoop user name. Here I set root. If it is another user name, it needs to be modified.
        Configuration config = new Configuration();
        //Declare a new access configuration object
        config.set("fs.defaultFS","hdfs://192.168.56.201:8020");
        //Set the specific address to access
        FileSystem fs = FileSystem.get(config);
        //Create a new file system object
        FileStatus[] stas = fs.listStatus(new Path("/"));
        for(FileStatus f : stas){<!-- -->
            System.out.println(f.getPermission().toString() + "" + f.getPath().toString());
            //Output all files or directories under the root directory, excluding subdirectories
        }
        fs.close();
    }
}

Output result:
2. Write content to HDFS writefiles
Reference Code:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.IOException;
import java.io.OutputStream;

public class Demo04WriteFile {<!-- -->
    public static void main(String[] args) throws IOException {<!-- -->
        String server = "hdfs://192.168.56.201:8020";
        System.setProperty("HADOOP_USER_NAME", "root");
        Configuration config = new Configuration();
        config.set("fs.defaultFS", server);
        try (FileSystem fs = FileSystem.get(config)) {<!-- -->
            OutputStream out = fs.create(new Path(server + "/test/b.txt"));
           out.write("Hello hadoop\\
".getBytes());
           out.write("Chinese writing test\\
".getBytes());
           out.close();
        }
    }
}

Enter hdfs dfs -cat /test/b.txt to query and write successfully

3.listfile displays all files
Reference Code:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;

import java.io.IOException;

public class Demo02ListFiles {<!-- -->
    public static void main(String[] args) throws IOException {<!-- -->
        System.setProperty("HADOOP_USER_NAME", "root");
        Configuration config = new Configuration();
        config.set("fs.defaultFS", "hdfs://192.168.56.201:8020");
        FileSystem fs = FileSystem.get(config);
        RemoteIterator<LocatedFileStatus> files =
                fs.listFiles(new Path("/test"), true);
        while (files.hasNext()) {<!-- -->
            LocatedFileStatus file = files.next();

            System.out.println(file.getPermission() + " " + file.getPath());
        }

        fs.close();
    }
}

Output result:
4. Read the contents of the HDFS file filesystem.open
Code:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.DataInputStream;
import java.io.IOException;

public class Demo03ReadFile {<!-- -->
    public static void main(String[] args) throws IOException {<!-- -->
        String server = "hdfs://192.168.56.201:8020";
        System.setProperty("HADOOP_USER_NAME", "root");
        Configuration config = new Configuration();
        config.set("fs.defaultFS", server);
        try (FileSystem fs = FileSystem.get(config)) {<!-- -->
            DataInputStream in = fs.open(new Path(server + "/test/b.txt"));
            int len = 0;
            byte[] bs = new byte[1024];
            while ((len = in.read(bs)) != -1) {<!-- -->
                String str = new String(bs, 0, len);
                System.out.print(str);
            }
        }
    }}

Output result: