Preface
Springboot implements breakpoint upload, resume upload, and instant upload.
The saving method provides local upload (stand-alone) and minio upload (can be clustered)
This article is mainly about the back-end implementation solution. The database persistence uses jpa.
1. Implementation ideas
The front-end generates file md5, and checks the file block upload progress or seconds based on md5.
Files that need to be uploaded in shards Upload sharded files
Upload the shards to the server after merging
2. Database table objects
illustrate:
AbstractDomainPd is a public field, such as id, creator, creation time, etc. You can modify it according to your own framework.
clientId application id is used to isolate different application attachments, not required
Attachment table: Attachment information uploaded successfully
@Entity
@Table(name = "gsdss_file", schema = "public")
@Data
public class AttachmentPO extends AbstractDomainPd<String> implements Serializable {<!-- -->
/**
* relative path
*/
private String path;
/**
* file name
*/
private String fileName;
/**
* File size
*/
private String size;
/**
* File MD5
*/
private String fileIdentifier;
}
Sharding information table: records the uploaded shard data of the current file
@Entity
@Table(name = "gsdss_file_chunk", schema = "public")
@Data
public class ChunkPO extends AbstractDomainPd<String> implements Serializable {<!-- -->
/**
* application id
*/
private String clientId;
/**
* File block number, starting from 1
*/
private Integer chunkNumber;
/**
* File identifier MD5
*/
private String fileIdentifier;
/**
* file name
*/
private String fileName;
/**
* relative path
*/
private String path;
}
2. Business input objects
Check file block upload progress or pass in seconds parameter object
package com.gsafety.bg.gsdss.file.manage.model.req;
import io.swagger.v3.oas.annotations.Hidden;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.springframework.web.multipart.MultipartFile;
import javax.validation.constraints.NotNull;
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class ChunkReq {<!-- -->
/**
* File block number, starting from 1
*/
@NotNull
private Integer chunkNumber;
/**
* File identifier MD5
*/
@NotNull
private String fileIdentifier;
/**
* relative path
*/
@NotNull
private String path;
/**
*Block content
*/
@Hidden
privateMultipartFile file;
/**
* application id
*/
@NotNull
private String clientId;
/**
* file name
*/
@NotNull
private String fileName;
}
Upload fragment input parameters
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class CheckChunkReq {<!-- -->
/**
* application id
*/
@NotNull
private String clientId;
/**
* file name
*/
@NotNull
private String fileName;
/**
*md5
*/
@NotNull
private String fileIdentifier;
}
Slice merge into parameters
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class FileReq {<!-- -->
@Hidden
privateMultipartFile file;
/**
* file name
*/
@NotNull
private String fileName;
/**
* File size
*/
@NotNull
private Long fileSize;
/**
*eg:data/plan/
*/
@NotNull
private String path;
/**
*md5
*/
@NotNull
private String fileIdentifier;
/**
* application id
*/
@NotNull
private String clientId;
}
Check file block upload progress or return results in seconds
@Data
public class UploadResp implements Serializable {<!-- -->
/**
* Whether to skip uploading (already uploaded items can be skipped directly to achieve the effect of instant upload)
*/
private boolean skipUpload = false;
/**
* The file block number that has been uploaded can be skipped and resumed at a breakpoint.
*/
private List<Integer> uploadedChunks;
/**
*File information
*/
private AttachmentResp fileInfo;
}
3. Local upload implementation
@Resource
private S3OssProperties properties;
@Resource
private AttachmentService attachmentService;
@Resource
private ChunkDao chunkDao;
@Resource
private ChunkMapping chunkMapping;
/**
* Upload fragmented files
*
* @param req
*/
@Override
public boolean uploadChunk(ChunkReq req) {<!-- -->
BizPreconditions.checkArgumentNoStack(!req.getFile().isEmpty(), "The uploaded fragment cannot be empty!");
BizPreconditions.checkArgumentNoStack(req.getPath().endsWith("/"), "url parameter must end with /");
//File name-1
String fileName = req.getFileName().concat("-").concat(req.getChunkNumber().toString());
//Directory address of the fragmented file upload server Folder address/chunks/file md5
String filePath = properties.getPath().concat(req.getClientId()).concat(File.separator).concat(req.getPath())
.concat("chunks").concat(File.separator).concat(req.getFileIdentifier()).concat(File.separator);
try {<!-- -->
Path newPath = Paths.get(filePath);
Files.createDirectories(newPath);
//Folder address/md5/file name-1
newPath = Paths.get(filePath.concat(fileName));
if (Files.notExists(newPath)) {<!-- -->
Files.createFile(newPath);
}
Files.write(newPath, req.getFile().getBytes(), StandardOpenOption.CREATE);
} catch (IOException e) {<!-- -->
log.error("Attachment storage failed", e);
throw new BusinessCheckException("Attachment storage failed");
}
//Storage shard information
chunkDao.save(chunkMapping.req2PO(req));
return true;
}
/**
* Check file blocks
*/
@Override
public UploadResp checkChunk(CheckChunkReq req) {<!-- -->
UploadResp result = new UploadResp();
//Query database records
//First determine whether the entire file has been uploaded. If so, tell the front end to skip uploading and achieve instant upload.
AttachmentResp resp = attachmentService.findByFileIdentifierAndClientId(req.getFileIdentifier(), req.getClientId());
if (resp != null) {<!-- -->
//Save current file information
AttachmentResp newResp = attachmentService.save(AttachmentReq.builder()
.fileName(req.getFileName()).origin(AttachmentConstants.TYPE.LOCAL_TYPE)
.clientId(req.getClientId()).path(resp.getPath()).size(resp.getSize())
.fileIdentifier(req.getFileIdentifier()).build());
result.setSkipUpload(true);
result.setFileInfo(newResp);
return result;
}
//If the complete file does not exist, go to the database to determine which file blocks have been uploaded, tell the result to the front end, skip the upload of these file blocks, and implement breakpoint resume uploading.
List<ChunkPO> chunkList = chunkDao.findByFileIdentifierAndClientId(req.getFileIdentifier(), req.getClientId());
//Return the chunkNumber list of existing chunks to the front end, and the front end will avoid these chunks
if (!CollectionUtils.isEmpty(chunkList)) {<!-- -->
List<Integer> collect = chunkList.stream().map(ChunkPO::getChunkNumber).collect(Collectors.toList());
result.setUploadedChunks(collect);
}
return result;
}
/**
* Merge shards
*
* @param req
*/
@Override
public boolean mergeChunk(FileReq req) {<!-- -->
String filename = req.getFileName();
String date = DateUtil.localDateToString(LocalDate.now());
//The attachment server stores the merged file storage address
String file = properties.getPath().concat(req.getClientId()).concat(File.separator).concat(req.getPath())
.concat(date).concat(File.separator).concat(filename);
//Server fragment file storage address
String folder = properties.getPath().concat(req.getClientId()).concat(File.separator).concat(req.getPath())
.concat("chunks").concat(File.separator).concat(req.getFileIdentifier());
//Merge files to local directory and delete fragmented files
boolean flag = mergeFile(file, folder, filename);
if (!flag) {<!-- -->
return false;
}
//Save file record
AttachmentResp resp = attachmentService.findByFileIdentifierAndClientId(req.getFileIdentifier(), req.getClientId());
if (resp == null) {<!-- -->
attachmentService.save(AttachmentReq.builder().fileName(filename).origin(AttachmentConstants.TYPE.LOCAL_TYPE)
.clientId(req.getClientId()).path(file).size(FileUtils.changeFileFormat(req.getFileSize()))
.fileIdentifier(req.getFileIdentifier()).build());
}
//After successfully inserting the file record, delete the corresponding record in the chunk table to free up space.
chunkDao.deleteAllByFileIdentifierAndClientId(req.getFileIdentifier(), req.getClientId());
return true;
}
/**
* File merge
*
* @param targetFile The file address to be formed
* @param folder Fragmented file storage address
* @param filename The name of the file
*/
private boolean mergeFile(String targetFile, String folder, String filename) {<!-- -->
try {<!-- -->
//First determine whether the file exists
if (FileUtils.fileExists(targetFile)) {<!-- -->
//File already exists
return true;
}
Path newPath = Paths.get(StringUtils.substringBeforeLast(targetFile, File.separator));
Files.createDirectories(newPath);
Files.createFile(Paths.get(targetFile));
Files.list(Paths.get(folder))
.filter(path -> !path.getFileName().toString().equals(filename))
.sorted((o1, o2) -> {<!-- -->
String p1 = o1.getFileName().toString();
String p2 = o2.getFileName().toString();
int i1 = p1.lastIndexOf("-");
int i2 = p2.lastIndexOf("-");
return Integer.valueOf(p2.substring(i2)).compareTo(Integer.valueOf(p1.substring(i1)));
})
.forEach(path -> {<!-- -->
try {<!-- -->
//Write to file in append form
Files.write(Paths.get(targetFile), Files.readAllBytes(path), StandardOpenOption.APPEND);
//Delete the block after merging
Files.delete(path);
} catch (IOException e) {<!-- -->
log.error(e.getMessage(), e);
throw new BusinessException("File merge failed");
}
});
//Delete empty folders
FileUtils.delDir(folder);
} catch (IOException e) {<!-- -->
log.error("File merge failed: ", e);
throw new BusinessException("File merge failed");
}
return true;
}
3. minio upload implementation
@Resource
private MinioTemplate minioTemplate;
@Resource
private AttachmentService attachmentService;
@Resource
private ChunkDao chunkDao;
@Resource
private ChunkMapping chunkMapping;
/**
* Upload fragmented files
*/
@Override
public boolean uploadChunk(ChunkReq req) {<!-- -->
String fileName = req.getFileName();
BizPreconditions.checkArgumentNoStack(!req.getFile().isEmpty(), "The uploaded fragment cannot be empty!");
BizPreconditions.checkArgumentNoStack(req.getPath().endsWith(separator), "url parameter must be / ends");
String newFileName = req.getPath().concat("chunks").concat(separator).concat(req.getFileIdentifier()).concat(separator)
+ fileName.concat("-").concat(req.getChunkNumber().toString());
try {<!-- -->
minioTemplate.putObject(req.getClientId(), newFileName, req.getFile());
} catch (Exception e) {<!-- -->
e.printStackTrace();
throw new BusinessException("File upload failed");
}
//Storage shard information
chunkDao.save(chunkMapping.req2PO(req));
return true;
}
/**
* Check file blocks
*/
@Override
public UploadResp checkChunk(CheckChunkReq req) {<!-- -->
UploadResp result = new UploadResp();
//Query database records
//First determine whether the entire file has been uploaded. If so, tell the front end to skip uploading and achieve instant upload.
AttachmentResp resp = attachmentService.findByFileIdentifierAndClientId(req.getFileIdentifier(), req.getClientId());
if (resp != null) {<!-- -->
//Save current file information
AttachmentResp newResp = attachmentService.save(AttachmentReq.builder()
.fileName(req.getFileName()).origin(AttachmentConstants.TYPE.MINIO_TYPE)
.clientId(req.getClientId()).path(resp.getPath()).size(resp.getSize())
.fileIdentifier(req.getFileIdentifier()).build());
result.setSkipUpload(true);
result.setFileInfo(newResp);
return result;
}
//If the complete file does not exist, go to the database to determine which file blocks have been uploaded, tell the result to the front end, skip the upload of these file blocks, and implement breakpoint resume uploading.
List<ChunkPO> chunkList = chunkDao.findByFileIdentifierAndClientId(req.getFileIdentifier(), req.getClientId());
//Return the chunkNumber list of existing chunks to the front end, and the front end will avoid these chunks
if (!CollectionUtils.isEmpty(chunkList)) {<!-- -->
List<Integer> collect = chunkList.stream().map(ChunkPO::getChunkNumber).collect(Collectors.toList());
result.setUploadedChunks(collect);
}
return result;
}
/**
* Merge shards
*
* @param req
*/
@Override
public boolean mergeChunk(FileReq req) {<!-- -->
String filename = req.getFileName();
//Merge files to local directory
String chunkPath = req.getPath().concat("chunks").concat(separator).concat(req.getFileIdentifier()).concat(separator);
List<Item> chunkList = minioTemplate.getAllObjectsByPrefix(req.getClientId(), chunkPath, false);
String fileHz = filename.substring(filename.lastIndexOf("."));
String newFileName = req.getPath() + UUIDUtil.uuid() + fileHz;
try {<!-- -->
List<ComposeSource> sourceObjectList = chunkList.stream()
.sorted(Comparator.comparing(Item::size).reversed())
.map(l -> ComposeSource.builder()
.bucket(req.getClientId())
.object(l.objectName())
.build())
.collect(Collectors.toList());
ObjectWriteResponse response = minioTemplate.composeObject(req.getClientId(), newFileName, sourceObjectList);
//Delete sharded buckets and files
minioTemplate.removeObjects(req.getClientId(), chunkPath);
} catch (Exception e) {<!-- -->
e.printStackTrace();
throw new BusinessException("File merge failed");
}
//Save file record
AttachmentResp resp = attachmentService.findByFileIdentifierAndClientId(req.getFileIdentifier(), req.getClientId());
if (resp == null) {<!-- -->
attachmentService.save(AttachmentReq.builder().fileName(filename).origin(AttachmentConstants.TYPE.MINIO_TYPE)
.clientId(req.getClientId()).path(newFileName).size(FileUtils.changeFileFormat(req.getFileSize()))
.fileIdentifier(req.getFileIdentifier()).build());
}
//After successfully inserting the file record, delete the corresponding record in the chunk table to free up space.
chunkDao.deleteAllByFileIdentifierAndClientId(req.getFileIdentifier(), req.getClientId());
return true;
}
MinioTemplate reference
Summarize
Check file upload progress in chunks or in seconds
Query the attachment information table according to the file md5, and if it exists, return the attachment information directly.
There is no query shard information table. Query the current file shard upload progress and return the shard number that has been uploaded.
Upload shards
The upload address of the fragmented file needs to be unique, and the file MD5 can be used as an isolation
Save multipart upload information after uploading
minio has a size limit for merging fragmented files. Except for the last fragment, the size of other fragmented files must not be less than 5MB. Therefore, minio fragmented upload requires a minimum fragment size of 5MB, and obtaining fragments needs to be sorted by fragment file size. , put the last fragment at the end and merge it
Shard merge
Merge the fragmented files into new files to the final file storage address and delete the fragmented files
Save the final file information to the attachment information table
Delete the corresponding shard information table data
Reference article: http://blog.ncmem.com/wordpress/2023/10/29/springboot-breakpoint upload, resume upload, and instant transfer implementation/
Welcome to join the group to discuss
