Preface
In many projects, you will encounter files such as uploading and downloading videos, update packages, and applications. The commonality of such files is that they are very large. In my projects, I have encountered files of about 4G that were downloaded by more than 100 machines at the same time. At this time, if It is impossible to upload and download using post, but Baidu search all said to adjust the post limit of php.ini, but this is a ridiculous solution, so another solution is needed – multipart upload. and download speed limit
Here I will show you how to use PHP to implement it. Various languages and frameworks are applicable at the same time. This time I used PHP’s laravel. The language and implementation ideas are the same.
If the multi-part upload is used in the project, I personally recommend finding the corresponding package such as (AetherUpload-Laravel), and if possible, directly use the multi-part upload service of large companies such as 7 Niu Cloud and Alibaba Cloud.
Multipart upload
principle
Divide the files that need to be uploaded into data blocks of the same size according to certain splitting rules;
Initialize a multipart upload task and return the unique identifier of this multipart upload;
Send each fragmented data block according to a certain strategy (serial or parallel);
After the transmission is completed, the server determines whether the uploaded data is complete. If it is complete, it synthesizes the data blocks to obtain the original file.
accomplish
h5
h5 implementation part. The h5 part realizes the division of files. During the upload, it tells the server the total number of files and the current file number. Each temporary file is sent out through an http request.
<!doctype html><html><head>
<meta charset="UTF-8">
<meta name="viewport"
content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>Document</title>
<style>
#progress{<!-- -->
width: 300px;
height: 20px;
background-color:#f7f7f7;
box-shadow:inset 0 1px 2px rgba(0,0,0,0.1);
border-radius:4px;
background-image:linear-gradient(to bottom,#f5f5f5,#f9f9f9);
}
#finish{<!-- -->
background-color: #149bdf;
background-image:linear-gradient(45deg,rgba(255,255,255,0.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,0.15) 50%,rgba(255,255,255,0.15) 75%,transparent 75%,transparent );
background-size:40px 40px;
display: inline-block;
height: 20px;
}
form{<!-- -->
margin-top: 50px;
}
</style></head><body><p id="progress">
<span id="finish" style="width: 0%;" progress="0"></span></p><form action="">
<input type="file" name="file" id="file">
<input type="button" value="stop" id="stop"></form><script>
var fileForm = document.getElementById("file");
var stopBtn = document.getElementById('stop');
var upload = new Upload();
fileForm.onchange = function(){<!-- -->
upload.addFileAndSend(this);
}
stopBtn.onclick = function(){<!-- -->
this.value = "Stop";
upload.stop();
this.value = "Stopped";
}
function Upload(){<!-- -->
var xhr = new XMLHttpRequest();
var form_data = new FormData();
const LENGTH = 1024 * 1024 *2;
var start = 0;
var end = start + LENGTH;
var blob;
var blob_num = 1;
var is_stop = 0
//External method, pass in the file object
this.addFileAndSend = function(that){<!-- -->
var file = that.files[0];
blob = cutFile(file);
sendFile(blob,file);
blob_num + = 1;
}
//Stop file upload
this.stop = function(){<!-- -->
xhr.abort();
is_stop = 1;
}
//cut file
function cutFile(file){<!-- -->
var file_blob = file.slice(start,end);
start = end;
end = start + LENGTH;
return file_blob;
};
//Send File
function sendFile(blob,file){<!-- -->
var form_data = new FormData();
var total_blob_num = Math.ceil(file.size / LENGTH);
form_data.append('file',blob);
form_data.append('blob_num',blob_num);
form_data.append('total_blob_num',total_blob_num);
form_data.append('file_name',file.name);
xhr.open('POST','http://vnn-admin.cc/Api/sliceUpload',false);
xhr.onreadystatechange = function () {<!-- -->
if (xhr.readyState==4 & amp; & amp; xhr.status==200)
{<!-- -->
console.log(xhr.responseText);
}
var progress;
var progressObj = document.getElementById('finish');
if(total_blob_num == 1){<!-- -->
progress = '100%';
}else{<!-- -->
progress = Math.min(100,(blob_num/total_blob_num)* 100 ) + '%';
// console.log(progress);
// console.log('Split');
}
progressObj.style.width = progress;
var t = setTimeout(function(){<!-- -->
if(start < file.size & amp; & amp; is_stop === 0){<!-- -->
blob = cutFile(file);
sendFile(blob,file);
blob_num + = 1;
}else{<!-- -->
setTimeout(t);
}
},1000);
}
xhr.send(form_data);
}
}</script></body></html>
Server
The server receives the uploaded file piece and determines whether it is the last piece. If so, it merges the file and deletes the uploaded file piece.
/**
* @Desc: slice upload
*
* @param Request $request
* @return mixed
*/
public function sliceUpload(Request $request)
{<!-- -->
$file = $request->file('file');
$blob_num = $request->get('blob_num');
$total_blob_num = $request->get('total_blob_num');
$file_name = $request->get('file_name');
$realPath = $file->getRealPath(); //The absolute path of the temporary file
//storage address
$path = 'slice/'.date('Ymd') ;
$filename = $path .'/'. $file_name . '_' . $blob_num;
//upload
$upload = Storage::disk('admin')->put($filename, file_get_contents($realPath));
//Determine whether it is the last block, if so, perform file synthesis and delete the file block
if($blob_num == $total_blob_num){<!-- -->
for($i=1; $i<= $total_blob_num; $i + + ){<!-- -->
$blob = Storage::disk('admin')->get($path.'/'. $file_name.'_'.$i);// Storage::disk('admin ')->append($path.'/'.$file_name, $blob); //This method cannot be used. The function will add 0X0A, which is the \\
newline character, to the existing file.
file_put_contents(public_path('uploads').'/'.$path.'/'.$file_name,$blob,FILE_APPEND);
}
//Delete file blocks after merging
for($i=1; $i<= $total_blob_num; $i + + ){<!-- -->
Storage::disk('admin')->delete($path.'/'. $file_name.'_'.$i);
}
}
if ($upload){<!-- -->
return $this->json(200, 'Upload successful');
}else{<!-- -->
return $this->json(0, 'Upload failed');
}
}
Download speed limit
principle
Limit output bytes per second
Turn off buffer caching
accomplish
public function sliceDownload()
{<!-- -->
$path = 'slice/'.date('Ymd') ;
$filename = $path .'/'. 'Jay Chou - Black Humor [mqms2].mp3' ;
//Get file resources
$file = Storage::disk('admin')->readStream($filename);
//Get file size
$fileSize = Storage::disk('admin')->size($filename);
header("Content-type:application/octet-stream");//Set the header to download
header("Accept-Ranges:bytes");
header("Accept-Length:".$fileSize);//Response size
header("Content-Disposition: attachment; filename=Jay Chou - Black Humor [mqms2].mp3");//File name
//If not set, it will wait until the buffer is full before responding.
ob_end_clean();//End of buffer
ob_implicit_flush();//Force to send the output to the browser immediately whenever there is output\
header('X-Accel-Buffering: no'); // No buffering of data
$limit=1024*1024;
$count=0;
//Limit the rate per second
while($fileSize-$count>0){<!-- -->//Loop to read file data
$data=fread($file,$limit);
$count + =$limit;
echo $data;//output file
sleep(1);
}
}
Just adjust the value of $limit when you need greater speed.
Summarize
So far, we have finished talking about the principle and simple implementation demo of multi-part upload and download speed limit. You should know how to realize multi-part upload. I hope it will be helpful to everyone, because large file upload and download are things that are often encountered in implementation.
Reference article: http://blog.ncmem.com/wordpress/2023/11/06/Large file transfer solution: multipart upload-download speed limit/
Welcome to join the group to discuss
