Downloading files from the Internet is a bit similar to copying a local file to another directory, and we also use IO streams to operate. For downloading from the Internet, you also need to establish a connection between the local computer and the server where the downloaded file is located.
3. Basic code of file downloader
3.1 HttpURLConnection
If you download a file from the Internet, you need to establish a connection with the server where the file is located. Here you can use the java.net.HttpURLConnection class provided by jdk to help us complete this operation. jdk11 provides the java.net.http.HttpClient class to replace HttpURLConnection. Since jdk8 is used now, the HttpClient in jdk11 is not needed for now. In addition, there are some other third-party provided classes that can perform similar operations, so I won’t go into details here.
3.2 User ID
When we access a website through a browser, the identification of the current browser version, operating system version and other information will be sent to the server where the website is located. When using program code to access a website, this identifier needs to be sent.
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.163 Safari/535.1
4. Download information
4.1 Scheduled tasks
When downloading a file, it is best to display the download speed, downloaded file size and other information. Here you can get the download information of the file at regular intervals, such as once every 1 second, and then print the information to the console. File downloading is an independent thread, and another thread needs to be opened to obtain file information at intervals. The java.util.concurrent.ScheduledExecutorService class can help us implement this function.
4.2 ScheduledExecutorService
This class provides some methods to help developers achieve the effect of interval execution. Some common methods and their parameter descriptions are listed below. We can get an object of this class in the following way, where 1 identifies the number of core threads.
ScheduledExecutorService s = Executors.newScheduledThreadPool(1);
schedule method
This method is overloaded. Both overloaded methods have 3 formal parameters, but the first formal parameter is different.
Parameters
Meaning
Runnable / Callable
These two types of tasks can be passed in
long delay
The amount of delay time
TimeUnit unit
Time unit
The function of this method is to delay the execution of the task according to the specified time.
scheduleAtFixedRate method
The function of this method is to delay execution according to the specified time and continue execution at regular intervals.
Parameters
Meaning
Runnable command
Executed tasks
long initialDelay
The number of delayed times
long period
long period
td>
The number of intervals
TimeUnit unit
Time unit
If the execution time of a task exceeds the interval, the task will be executed again directly after the execution is completed, instead of waiting for the interval to be executed.
scheduleWithFixedDelay method
The function of this method is to delay execution according to the specified time and continue execution at regular intervals.
Parameters
Meaning
Runnable command
Executed tasks
long initialDelay
The number of delayed times
long period
long period
td>
The number of intervals
TimeUnit unit
Time unit
When executing a task, no matter how long it takes, after the task is executed, it will wait for the interval before continuing the next task.
5. Introduction to thread pool
Threads will consume some resources during the process of creation and destruction. In order to save these costs, jdk added a thread pool. The thread pool saves overhead and improves the efficiency of thread usage. Alibaba development documentation recommends using a thread pool when writing multi-threaded programs.
5.1 ThreadPoolExecutor constructor parameters
The ThreadPoolExecutor class is provided under the juc package, through which a thread pool can be created. There are 4 overloaded constructors in this class. The core constructor has 7 formal parameters. The meanings of these parameters are as follows: :
Parameter
Meaning
corePoolSize
The number of core threads in the thread pool
maximumPoolSize
The maximum number of threads in the thread pool is the sum of the number of core threads and the number of non-core threads
td>
keepAliveTime
The idle survival time of non-core threads
unit
keepAliveTime The survival time unit
workQueue
When there are no idle threads, new tasks will be added to the workQueue and queued up to wait
threadFactory
Thread factory, used to create threads
handler
Rejection policy, when the task is too Rejection strategy when too many cannot be processed
5.2 Thread pool working process
5.3 Thread pool status
Status
Description
RUNNING
The state after creating the thread pool is RUNNING
SHUTDOWN
In this state, the thread pool will not receive new tasks, but will process the remaining tasks in the blocking queue. , relatively mild
STOP
In this state, the executing task will be interrupted and the blocked queue task will be abandoned, relatively violent
TIDYING
All tasks have been executed, the active thread is 0 and it is about to enter the termination
TERMINATED
thread Pool termination
5.4 Closing the thread pool
The thread pool needs to be closed after use. The following two methods are provided for closing.
Method
Description
shutdown()
After this method is executed, the thread pool status changes to SHUTDOWN, and new tasks will not be received, but submitted tasks will be executed. This method will not block the execution of the calling thread.
shutdownNow()
After this method is executed, the thread pool status changes to STOP, no new tasks will be received, and the tasks in the queue will be returned. And use interrupt to interrupt the executing task.
5.5 Work Queue
Some work queues workQueue provided in jdk.
Queue
Description
SynchronousQueue
Submit the queue directly
ArrayBlockingQueue
Bounded queue, you can specify the capacity
LinkedBlockingDeque
Unbounded Queue
PriorityBlockingQueue
Priority task queue, which can execute tasks according to task priority order
6. Code implementation
6.1 Environment setup
Basic information
Development tools: IDEA
JDK version: 8
Project encoding: utf-8
Create project
Just create a javase project in the development tool without importing third-party jar dependencies.
6.2 Implementation logic
First determine whether duplicate files already exist. This step can actually be ignored, because the final downloaded and merged file name has been uniquely identified by a timestamp;
Start a thread to print the download status every second;
Divide tasks into multiple threads for fast downloading;
After all block files are downloaded, the block files are merged;
After merging the chunked files, clean the chunked files;
Release resources and close the thread pool and connection object.
6.3 Project Structure
Package name
Function
constant
Package that stores constant classes
core
Package that stores downloader core classes
util
Package for storing tool classes
Main
Main class
< /table>
6.4 Class Code
constant package
Constant
/**
* Description: Store project constants
*
* @Author Fox half face Tim
* @Create 2023/11/6 1:22
* @Version 1.0
*/
public class Constant {<!-- -->
/**
* Specify the storage location of the download directory
*/
public static final String PATH = "D:\download";
public static final double MB = 1024d * 1024d;
public static final double KB = 1024d;
/**
* The byte size of each read
*/
public static final int BYTE_SIZE = 1024 * 100;
/**
* Suffix for block files (temporary files)
*/
public static final String PART_FILE_SUFFIX = ".temp";
/**
* Number of threads
*/
public static final int THREAD_NUM = 5;
//Code to create storage location
// public static void main(String[] args) {<!-- -->
// File file = new File("D:\download");
// if (!file.exists()) {<!-- -->
// file.mkdir();
// }
// }
}
util package
FileUtils
/**
* Description: File related tools
*
* @Author Fox half face Tim
* @Create 2023/11/6 11:46
* @Version 1.0
*/
public class FileUtils {<!-- -->
/**
* Get the size of local files
*
* @param path file path
* @return file size
*/
public static long getFileContentLength(String path) {<!-- -->
File file = new File(path);
return file.exists() & amp; & amp; file.isFile() ? file.length() : 0;
}
}
HttpUtils
/**
* Description: Http related tools
*
* @Author Fox half face Tim
* @Create 2023/11/6 1:06
* @Version 1.0
*/
public class HttpUtils {<!-- -->
private static long id = System.currentTimeMillis();
public static void change() {<!-- -->
id = System.currentTimeMillis();
}
/**
* Get downloaded file size
*
* @param url download file link
* @return file size
* @throwsIOException
*/
public static long getHttpFileContentLength(String url) throws IOException {<!-- -->
int contentLength;
HttpURLConnection httpURLConnection = null;
try {<!-- -->
httpURLConnection = getHttpURLConnection(url);
contentLength = httpURLConnection.getContentLength();
} finally {<!-- -->
if (httpURLConnection != null) {<!-- -->
httpURLConnection.disconnect();
}
}
return contentLength;
}
/**
* Download in chunks
*
* @param url download address
* @param startPos starting position of downloaded file
* @param endPos end position of downloaded file
* @return connection object
*/
public static HttpURLConnection getHttpURLConnection(String url, long startPos, long endPos) throws IOException {<!-- -->
HttpURLConnection httpURLConnection = getHttpURLConnection(url);
LogUtils.info("The download range is: {}-{}", startPos, endPos);
if (endPos != 0) {<!-- -->
httpURLConnection.setRequestProperty("RANGE", "bytes=" + startPos + "-" + endPos);
} else {<!-- -->
httpURLConnection.setRequestProperty("RANGE", "bytes=" + startPos + "-");
}
return httpURLConnection;
}
/**
* Get the HttpURLConnection connection object
*
* @param url file address
* @return HttpURLConnection connection object
*/
public static HttpURLConnection getHttpURLConnection(String url) throws IOException {<!-- -->
URL httpUrl = new URL(url);
HttpURLConnection httpURLConnection = (HttpURLConnection) httpUrl.openConnection();
//Send identification information to the server where the file is located
httpURLConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.163 Safari/535.1");
return httpURLConnection;
}
/**
* Get the name of the downloaded file
*
* @param url download address
* @return file name
*/
public static String getHttpFileName(String url) {<!-- -->
String fileName;
int startIndex = url.lastIndexOf("/");
int endIndex = url.lastIndexOf("?");
if (endIndex == -1) {<!-- -->
fileName = url.substring(startIndex + 1);
} else {<!-- -->
fileName = url.substring(startIndex + 1, endIndex);
}
int pointIndex = fileName.lastIndexOf(".");
return fileName.substring(0, fileName.lastIndexOf(".")) + "-" + id + fileName.substring(pointIndex);
}
}
/**
* Description: Display download information
*
* @Author Fox half face Tim
* @Create 2023/11/6 2:07
* @Version 1.0
*/
@SuppressWarnings("AlibabaUndefineMagicConstant")
public class DownloadInfoThread implements Runnable {<!-- -->
/**
*Total download file size
*/
private final long httpFileContentLength;
/**
* The cumulative download size this time
*/
public static volatile LongAdder downSize = new LongAdder();
/**
* The size of the previous download
*/
public double prevSize;
public DownloadInfoThread(long httpFileContentLength) {<!-- -->
this.httpFileContentLength = httpFileContentLength;
}
@Override
public void run() {<!-- -->
// Calculate the total file size in MB
String httpFileSize = String.format("%.2f", httpFileContentLength / Constant.MB);
// Calculate download speed kb per second
int speed = (int) ((downSize.doubleValue() - prevSize) / Constant.KB);
prevSize = downSize.doubleValue();
//The size of the remaining files
double remainSize = httpFileContentLength - downSize.doubleValue();
// Calculate remaining time
String remainTime = String.format("%.1f", remainSize / Constant.KB / speed);
if ("Infinity".equalsIgnoreCase(remainTime)) {<!-- -->
remainTime = "-";
}
//Downloaded size
String currentFileSize = String.format("%.1f", downSize.doubleValue() / Constant.MB);
String speedInfo = String.format("Downloaded %smb/%smb, speed %skb/s, remaining time %ss", currentFileSize, httpFileSize, speed, remainTime);
System.out.print("\r");
System.out.print(speedInfo);
}
}
DownloaderTask
/**
* Description: Chunked download task
*
* @Author Fox half face Tim
* @Create 2023/11/7 0:58
* @Version 1.0
*/
public class DownloaderTask implements Callable<Boolean> {<!-- -->
private final String url;
/**
* Download starting position
*/
private final long startPos;
/**
* Download end position
*/
private final long endPos;
/**
* Identifies which part the current part is
*/
private final int part;
private final CountDownLatch countDownLatch;
public DownloaderTask(String url, long startPos, long endPos, int part, CountDownLatch countDownLatch) {<!-- -->
this.url = url;
this.startPos = startPos;
this.endPos = endPos;
this.part = part;
this.countDownLatch = countDownLatch;
}
@Override
public Boolean call() throws Exception {<!-- -->
// Get file name
String httpFileName = HttpUtils.getHttpFileName(url);
// Chunked file name
httpFileName = httpFileName + Constant.PART_FILE_SUFFIX + part;
//Download path
httpFileName = Constant.PATH + httpFileName;
// Get the connection for downloading in chunks
HttpURLConnection httpURLConnection = HttpUtils.getHttpURLConnection(url, startPos, endPos);
try (
InputStream input = httpURLConnection.getInputStream();
BufferedInputStream bis = new BufferedInputStream(input);
RandomAccessFile accessFile = new RandomAccessFile(httpFileName, "rw");
) {<!-- -->
byte[] buffer = new byte[Constant.BYTE_SIZE];
int len;
// Loop to read data
while ((len = bis.read(buffer)) != -1) {<!-- -->
//Data downloaded within 1s is downloaded through the atomic class
DownloadInfoThread.downSize.add(len);
accessFile.write(buffer, 0, len);
}
} catch (FileNotFoundException e) {<!-- -->
LogUtils.error("Download file does not exist {}", url);
return false;
} catch (Exception e) {<!-- -->
LogUtils.error("Exception occurred during download");
return false;
} finally {<!-- -->
httpURLConnection.disconnect();
countDownLatch.countDown();
}
return true;
}
}
Downloader
/**
* Description: Downloader
*
* @Author Fox half face Tim
* @Create 2023/11/6 1:21
* @Version 1.0
*/
public class Downloader {<!-- -->
private final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(1);
public ThreadPoolExecutor poolExecutor = new ThreadPoolExecutor(Constant.THREAD_NUM,
Constant.THREAD_NUM,
0,
TimeUnit.SECONDS,
new ArrayBlockingQueue<>(5));
private CountDownLatch countDownLatch = new CountDownLatch(Constant.THREAD_NUM);
public void download(String url) {<!-- -->
// Get file name
String httpFileName = HttpUtils.getHttpFileName(url);
//File download path
httpFileName = Constant.PATH + httpFileName;
// Get the size of the local file
long localFileLength = FileUtils.getFileContentLength(httpFileName);
HttpURLConnection httpURLConnection = null;
DownloadInfoThread downloadInfoThread;
try {<!-- -->
// Get the connection object
httpURLConnection = HttpUtils.getHttpURLConnection(url);
// Get the total size of downloaded files
int contentLength = httpURLConnection.getContentLength();
// Determine whether the file has been downloaded
if (localFileLength >= contentLength) {<!-- -->
LogUtils.info("{} has been downloaded, no need to re-download", httpFileName);
//Close the connection object
httpURLConnection.disconnect();
// Close the thread pool
scheduledExecutorService.shutdownNow();
poolExecutor.shutdown();
return;
}
//Create a task object to obtain download information
downloadInfoThread = new DownloadInfoThread(contentLength);
// Give the task to the thread for execution and print every 1s
scheduledExecutorService.scheduleAtFixedRate(downloadInfoThread, 1, 1, TimeUnit.SECONDS);
// Split tasks
ArrayList<Future> list = new ArrayList<>();
split(url, list);
countDownLatch.await();
System.out.print("\r");
System.out.println("Blocked file download completed");
// merge files
if (merge(httpFileName)) {<!-- -->
// Clear temporary files
clearTemp(httpFileName);
}
} catch (IOException | InterruptedException e) {<!-- -->
e.printStackTrace();
} finally {<!-- -->
System.out.println("This execution is completed");
// Close the connection object
if (httpURLConnection != null) {<!-- -->
httpURLConnection.disconnect();
}
// Close the thread pool
scheduledExecutorService.shutdownNow();
poolExecutor.shutdown();
}
}
/**
*File splitting
*
* @param url file link
* @param futureList task collection
*/
public void split(String url, ArrayList<Future> futureList) {<!-- -->
try {<!-- -->
// Get download file size
long contentLength = HttpUtils.getHttpFileContentLength(url);
// Calculate the file size after splitting
long size = contentLength / Constant.THREAD_NUM;
// Calculate the number of blocks
for (int i = 0; i < Constant.THREAD_NUM; i + + ) {<!-- -->
// Calculate download starting position
long startPos = i * size;
// Calculate the end position
long endPos;
if (i == Constant.THREAD_NUM - 1) {<!-- -->
// Download the last block
endPos = 0;
} else {<!-- -->
endPos = startPos + size - 1;
}
//Create task object
DownloaderTask downloaderTask = new DownloaderTask(url, startPos, endPos, i, countDownLatch);
// Submit the task to the thread pool
Future<Boolean> future = poolExecutor.submit(downloaderTask);
futureList.add(future);
}
} catch (IOException e) {<!-- -->
throw new RuntimeException(e);
}
}
/**
* File merge
*
* @param fileName file name
* @return Whether the merge is successful
*/
public boolean merge(String fileName) {<!-- -->
LogUtils.info("Start merging files {}", fileName);
byte[] buffer = new byte[Constant.BYTE_SIZE];
int len;
try (
RandomAccessFile accessFile = new RandomAccessFile(fileName, "rw")
) {<!-- -->
for (int i = 0; i < Constant.THREAD_NUM; i + + ) {<!-- -->
try (
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(fileName + Constant.PART_FILE_SUFFIX + i))
) {<!-- -->
while ((len = bis.read(buffer)) != -1) {<!-- -->
accessFile.write(buffer, 0, len);
}
}
}
LogUtils.info("File merge completed {}", fileName);
} catch (Exception e) {<!-- -->
e.printStackTrace();
return false;
}
return true;
}
/**
* Clear temporary files
*
* @param fileName file name
*/
public void clearTemp(String fileName) {<!-- -->
LogUtils.info("Clean chunked files");
for (int i = 0; i < Constant.THREAD_NUM; i + + ) {<!-- -->
String name = fileName + Constant.PART_FILE_SUFFIX + i;
File file = new File(name);
file.delete();
}
LogUtils.info("Block clearing completed");
}
}
Main main class
public class Main {<!-- -->
public static void main(String[] args) {<!-- -->
//Create a directory to store the downloaded installation package. If it exists, it will not be created repeatedly.
createDir();
// download link
String url = null;
if (args == null || args.length == 0) {<!-- -->
while (url == null || url.trim().isEmpty()) {<!-- -->
System.out.print("Please enter the download link:");
Scanner scanner = new Scanner(System.in);
url = scanner.next();
}
} else {<!-- -->
url = args[0];
}
Downloader downloader = new Downloader();
downloader.download(url);
}
public static void createDir() {<!-- -->
File file = new File("D:\download");
if (!file.exists()) {<!-- -->
file.mkdir();
}
}
}
6.5 Code Test
Test download link-QQ: https://dldir1.qq.com/qqfile/qq/QQNT/1e2b98d8/QQ9.9.3.17816_x64.exe
After the download is completed, we can find the downloaded qq installation package in the D:\download directory.
