File upload is often encountered in web development
The default configuration of springboot is 10MB. Anything larger than 10M cannot be uploaded to the server. You need to modify the default configuration.
However, if it is modified to support large files, it will increase the burden on the server.
When the file is larger than a certain level, not only will the server take up a lot of memory, but the http transmission will most likely be interrupted.
Can be uploaded by cutting into parts
The file API provided by html5 can easily divide and slice files, and then transmit data to the server through ajax asynchronous processing, breaking through the restrictions on large file uploads. At the same time, asynchronous processing also improves the efficiency of file uploading to a certain extent.
Process description:
Split the file into N pieces
To process sharding, the frontend will call the upload interface multiple times, and each time a part of the file will be uploaded to the server.
After all N fragments are uploaded, merge the N files into one file and delete the N fragment files.
1. Server
(1) Add dependencies
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.3.3</version>
</dependency>
(2)UploadController
package com.example.demo.controller;
import com.example.demo.core.Result;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.io.FileUtils;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.CrossOrigin;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.Date;
@CrossOrigin
@Controller
@RequestMapping("/api/upload")
public class UploadController {
@PostMapping("/part")
@ResponseBody
public Result bigFile(HttpServletRequest request, HttpServletResponse response, String guid, Integer chunk, MultipartFile file, Integer chunks) {
try {
String projectUrl = System.getProperty("user.dir").replaceAll("\", "/");
;
boolean isMultipart = ServletFileUpload.isMultipartContent(request);
if (isMultipart) {
if (chunk == null) chunk = 0;
//Temporary directory is used to store all fragmented files
String tempFileDir = projectUrl + "/upload/" + guid;
File parentFileDir = new File(tempFileDir);
if (!parentFileDir.exists()) {
parentFileDir.mkdirs();
}
// During shard processing, the frontend will call the upload interface multiple times, and each time a part of the file will be uploaded to the backend.
File tempPartFile = new File(parentFileDir, guid + "_" + chunk + ".part");
FileUtils.copyInputStreamToFile(file.getInputStream(), tempPartFile);
}
} catch (Exception e) {
return Result.failMessage(400,e.getMessage());
}
return Result.successMessage(200,"Last success");
}
@RequestMapping("merge")
@ResponseBody
public Result mergeFile(String guid, String fileName) {
// Get destTempFile which is the final file
String projectUrl = System.getProperty("user.dir").replaceAll("\", "/");
try {
String sname = fileName.substring(fileName.lastIndexOf("."));
//Time formatting format
Date currentTime = new Date();
SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyyMMddHHmmssSSS");
//Get the current time and use it as a timestamp
String timeStamp = simpleDateFormat.format(currentTime);
//Splice new file name
String newName = timeStamp + sname;
simpleDateFormat = new SimpleDateFormat("yyyyMM");
String path = projectUrl + "/upload/";
String tmp = simpleDateFormat.format(currentTime);
File parentFileDir = new File(path + guid);
if (parentFileDir.isDirectory()) {
File destTempFile = new File(path + tmp, newName);
if (!destTempFile.exists()) {
//First get the upper-level directory of the file, create the upper-level directory, and then create the file
destTempFile.getParentFile().mkdir();
try {
destTempFile.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
}
for (int i = 0; i < parentFileDir.listFiles().length; i + + ) {
File partFile = new File(parentFileDir, guid + "_" + i + ".part");
FileOutputStream destTempfos = new FileOutputStream(destTempFile, true);
//Traverse "all fragmented files" to the "final file"
FileUtils.copyFile(partFile, destTempfos);
destTempfos.close();
}
// Delete the fragment files in the temporary directory
FileUtils.deleteDirectory(parentFileDir);
return Result.successMessage(200,"Merge successful");
}else{
return Result.failMessage(400,"Directory not found");
}
} catch (Exception e) {
return Result.failMessage(400,e.getMessage());
}
}
}
illustrate:
Annotate @CrossOrigin to solve cross-domain issues
(3)Result
package com.example.demo.core;
import com.alibaba.fastjson.JSON;
/**
* Created by Beibei on 19/02/22
*API response result
*/
public class Result<T> {
private int code;
private String message;
private T data;
public Result setCode(Integer code) {
this.code = code;
return this;
}
public int getCode() {
return code;
}
public String getMessage() {
return message;
}
public Result setMessage(String message) {
this.message = message;
return this;
}
public T getData() {
return data;
}
public Result setData(T data) {
this.data = data;
return this;
}
@Override
public String toString() {
return JSON.toJSONString(this);
}
public static <T> Result<T> fail(Integer code,T data) {
Result<T> ret = new Result<T>();
ret.setCode(code);
ret.setData(data);
return ret;
}
public static <T> Result<T> failMessage(Integer code,String msg) {
Result<T> ret = new Result<T>();
ret.setCode(code);
ret.setMessage(msg);
return ret;
}
public static <T> Result<T> successMessage(Integer code,String msg) {
Result<T> ret = new Result<T>();
ret.setCode(code);
ret.setMessage(msg);
return ret;
}
public static <T> Result<T> success(Integer code,T data) {
Result<T> ret = new Result<T>();
ret.setCode(code);
ret.setData(data);
return ret;
}
}
2. Front end
(1) Use plug-ins
webuploader, download https://github.com/fex-team/webuploader/releases
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<link href="css/webuploader.css" rel="stylesheet" type="text/css" />
<script type="text/javascript" src="jquery-1.10.1.min.js"></script>
<script type="text/javascript" src="dist/webuploader.min.js"></script>
</head>
<body>
<div id="uploader">
<div class="btns">
<div id="picker">Select file</div>
<button id="startBtn" class="btn btn-default">Start upload</button>
</div>
</div>
</body>
<script type="text/javascript">
var GUID = WebUploader.Base.guid();//A GUID
var uploader = WebUploader.create({
// swf file path
swf: 'dist/Uploader.swf',
// File receiving server.
server: 'http://localhost:8080/api/upload/part',
formData:{
guid: GUID
},
pick: '#picker',
chunked : true, // chunking processing
chunkSize: 1 * 1024 * 1024, // 1M per piece,
chunkRetry: false,//If it fails, do not retry
threads: 1, // Number of concurrent uploads. The maximum number of simultaneous upload processes allowed.
resize: false
});
$("#startBtn").click(function () {
uploader.upload();
});
//Triggered when file upload is successful.
uploader.on( "uploadSuccess", function( file ) {
$.post('http://localhost:8080/api/upload/merge', { guid: GUID, fileName: file.name}, function (data) {
if(data.code == 200){
alert('Upload successful!');
}
});
});
</script>
</html>
(2) Do not use plug-ins
Directly use HTML5 File API
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<script src="jquery-1.10.1.min.js" type="text/javascript">
</script>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
</head>
<body>
<div id="uploader">
<div class="btns">
<input id="file" name="file" type="file"/>
<br>
<br>
<button id="startBtn">
Start upload
</button>
</br>
</br>
</div>
<div id="output">
</div>
</div>
</body>
<script type="text/javascript">
var status = 0;
var page = {
init: function(){
$("#startBtn").click($.proxy(this.upload, this));
},
upload: function(){
status = 0;
varGUID = this.guid();
var file = $("#file")[0].files[0], //File object
name = file.name, //file name
size = file.size; //total size
var shardSize = 20 * 1024 * 1024, //use 1MB as a shard
shardCount = Math.ceil(size / shardSize); //Total number of slices
for(var i = 0;i < shardCount; + + i){
//Calculate the starting and ending positions of each slice
var start = i * shardSize,
end = Math.min(size, start + shardSize);
var partFile = file.slice(start,end);
this.partUpload(GUID,partFile,name,shardCount,i);
}
},
partUpload:function(GUID,partFile,name,chunks,chunk){
//Construct a form. FormData is new to HTML5.
var now = this;
var form = new FormData();
form.append("guid", GUID);
form.append("file", partFile); //slice method is used to cut out a part of the file
form.append("fileName", name);
form.append("chunks", chunks); //Total number of pieces
form.append("chunk", chunk); //Which piece is the current one?
//Ajax submission
$.ajax({
url: "http://localhost:8080/api/upload/part",
type: "POST",
data: form,
async: true, //asynchronous
processData: false, //very important, tell jquery not to process the form
contentType: false, //very important, specify false to form the correct Content-Type
success: function(data){
status + + ;
if(data.code == 200){
$("#output").html(status + " / " + chunks);
}
if(status==chunks){
now.mergeFile(GUID,name);
}
}
});
},
mergeFile:function(GUID,name){
var formMerge = new FormData();
formMerge.append("guid", GUID);
formMerge.append("fileName", name);
$.ajax({
url: "http://localhost:8080/api/upload/merge",
type: "POST",
data: formMerge,
processData: false, //very important, tell jquery not to process the form
contentType: false, //very important, specify false to form the correct Content-Type
success: function(data){
if(data.code == 200){
alert('Upload successful!');
}
}
});
},
guid:function(prefix){
var counter = 0;
var guid = ( + new Date()).toString( 32 ),
i = 0;
for ( ; i < 5; i + + ) {
guid + = Math.floor( Math.random() * 65535 ).toString( 32 );
}
return (prefix || 'wu_') + guid + (counter + + ).toString( 32 );
}
};
$(function(){
page.init();
});
</script>
</html>
3. Optimization
The default configuration of springboot is 10MB. When the front-end sharding is changed to 20M, an error will be reported.
org.apache.tomcat.util.http.fileupload.FileUploadBase$SizeLimitExceededException: the request was rejected because its size (10486839) exceeds the configured maximum (10485760)
Solution:
Add it to application.properties under src/main/resources
spring.servlet.multipart.max-file-size=30MB
spring.servlet.multipart.max-request-size=35MB
illustrate:
It is best to set a value larger than the value passed from the front end, otherwise it will not easily report an error.
Reference article: http://blog.ncmem.com/wordpress/2023/09/24/spring-boot implements segmented upload/
Welcome to join the group to discuss
