Article directory
- Column introduction
- 1. What data needs to be managed?
- 2. How to manage data
- 3. Design and implementation of data information structure
- 4.Data management design
- 5. Data management class implementation
- 6. Data management module organization
Introduction to the column
About the author: Hua Xiangyun, undergraduate student, C /C++ Rising Star Creator, Rising Star Program Mentor, Alibaba Cloud Expert Blogger, CSDN Content Partner… Dedicated to C/C++ and Linux learning.
Column introduction: This article is included in theC++ project – cloud backup
Related column recommendations: C Language Beginner Series, C Language Advanced Series< /em>, C++ Series, Data Structures and Algorithms, Linux< /strong>
Project Gitee link: https://gitee.com/li-yuanjiu/cloud-backup
1. What data needs to be managed
The actual storage path of the file
: When the customer wants to download a file, the data is read from this file to respond;File is compressed flag
: Determine whether the file has been compressed;Compressed package storage path
: If a file is a non-hotspot file, it will be compressed. This is the path where the compressed package is stored;- If the client wants to download a file, it needs to decompress it first and then read the decompressed file data;
There are also various types of file attribute information
:
File size
;File last modified time
;File last access time
;Resource path path in file access URL
;
2. How to manage data
-
For data information access
: Usehash
table to manage data in memory, use file accessURL
askey
, Thedata information structure
isval
. Access is fast. File accessURL
is used as thekey
because the client browser always usesURL
as the request when downloading files; -
Persistent storage management
: UseJSON serialization
to save all data to files;
3. Design and implementation of data information structure
- The
struct BackupInfo
structure contains various attribute information of the files to be managed; NewBackUpInfo
is responsible for organizing the attribute information of the passed in file;
typedef struct BackupInfo {<!-- --> bool pack_flag; // Whether the file is compressed flag size_t fsize; // file size time_t mtime; //The last modification time of the file time_t atime; //The last access time of the file std::string real_path; //The actual storage path of the file std::string pack_path; // Compressed package storage path std::string url; // File access URL bool NewBackupInfo(const std::string & amp;realpath) {<!-- --> FileUtil fu(realpath); if(fu.Exists() == false) {<!-- --> std::cout << "new backupinfo: file not exit" << std::endl; return false; } Config* config = Config::GetInstance(); std::string packdir = config->GetPackDir(); std::string packsuffix = config->GetPackFileSuffix(); std::string download_prefix = config->GetDownloadPrefix(); pack_flag = false; fsize = fu.FileSize(); mtime = fu.LastMtime(); atime = fu.LastATime(); real_path = realpath; pack_path = packdir + fu.FileName() + packsuffix; url = download_prefix + fu.FileName(); return true; } }BackupInfo;
4. Data management design
The data management class is responsible for managing data information.
Data management DataManger
includes the following members:
class DataManager {<!-- --> public: DataManager(); ~DataManager(); bool Insert(const BackupInfo & amp;info); // New bool Updata(const BackupInfo & amp;info); // Update bool GetOneByURL(const std::string & amp;url, BackupInfo* info); // Get file information through url bool GetOneByRealpath(const std::string & amp;realpath, BackupInfo* info); // Get file information through the actual path bool GetAll(std::vector<BackupInfo>* array); // Get all file information bool Storage(); // Every time data is updated or added, it must be stored persistently bool InitLoad(); // Initial loading, previous data must be loaded before each system startup private: std::string _backup_file; // Persistent storage of backup information pthread_rwlock_t _rwlock; // read-write lock std::unordered_map<std::string, BackupInfo> _table // Data information organization; };
5. Data management class implementation
class DataManager {<!-- --> public: DataManager() {<!-- --> _backup_file = Config::GetInstance()->GetBackupFile(); // Get the file for persistent storage of backup information pthread_rwlock_init( & amp;_rwlock, NULL); // Initialize read-write lock InitLoad(); // Initial loading } ~DataManager() {<!-- --> pthread_rwlock_destroy( & amp;_rwlock); // Release the read-write lock } bool Insert(const BackupInfo & amp;info) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); _table[info.url] = info; pthread_rwlock_unlock( & amp;_rwlock); return true; } bool Updata(const BackupInfo & amp;info) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); _table[info.url] = info; pthread_rwlock_unlock( & amp;_rwlock); return true; } bool GetOneByURL(const std::string & amp;url, BackupInfo* info) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); auto it = _table.find(url); if(it == _table.end()) {<!-- --> pthread_rwlock_unlock( & amp;_rwlock); return false; } *info = it->second; pthread_rwlock_unlock( & amp;_rwlock); return true; } bool GetOneByRealpath(const std::string & realpath, BackupInfo* info) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); auto it = _table.begin(); for(; it != _table.end(); + + it) {<!-- --> if(it->second.real_path == realpath) {<!-- --> *info = it->second; pthread_rwlock_unlock( & amp;_rwlock); return true; } } pthread_rwlock_unlock( & amp;_rwlock); return false; } bool GetAll(std::vector<BackupInfo>* array) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); auto it = _table.begin(); for(; it != _table.end(); + + it) {<!-- --> array->push_back(it->second); } pthread_rwlock_unlock( & amp;_rwlock); return true; } boolStorage() {<!-- --> // 1. Get all data std::vector<BackupInfo> array; GetAll(&array); // 2. Add to Json::Value Json::Value root; for(int i = 0; i < array.size(); i + + ) {<!-- --> Json::Value item; root["pack_flag"] = array[i].pack_flag; root["fsize"] = (Json::Int64)array[i].fsize; root["atime"] = (Json::Int64)array[i].atime; root["mtime"] = (Json::Int64)array[i].mtime; root["real_path"] = array[i].real_path; root["pack_path"] = array[i].pack_path; root["url"] = array[i].url; root.append(item); //Add array elements } // 3. Serialize Json::Value std::string body; JsonUtil::Serialize(root, & amp;body); // 4. Write file FileUtil fu(_backup_file); fu.SetContent(body); return true; } boolInitLoad() {<!-- --> // 1. Read the data from the file FileUtil fu(_backup_file); if(fu.Exists() == false) {<!-- --> return true; } std::string body; fu.GetContent( & amp;body); // 2. Deserialization Json::Value root; JsonUtil::Unserialize(body, & amp;root); // 3. Add the data in Json::Value obtained by deserialization to the table for(int i = 0; i < root.size(); i + + ) {<!-- --> BackupInfo info; info.pack_flag = root[i]["pack_flag"].asBool(); info.fsize = root[i]["fsize"].asInt64(); info.atime = root[i]["atime"].asInt64(); info.mtime = root[i]["mtime"].asInt64(); info.pack_path = root[i]["pack_path"].asString(); info.real_path = root[i]["real_path"].asString(); info.url = root[i]["url"].asString(); Insert(info); } return true; } private: std::string _backup_file; // Persistent storage of backup information pthread_rwlock_t _rwlock; // read-write lock std::unordered_map<std::string, BackupInfo> _table; // Data information organization };
6. Data management module organization
We put the implementations of BackpInfo
and DataManger
into data.hpp
.
// data.hpp #ifndef __MY_DATA__ #define __MY_DATA__ #include <unordered_map> #include <pthread.h> #include "util.hpp" #include "config.hpp" namespace cloud {<!-- --> typedef struct BackupInfo {<!-- --> bool pack_flag; // Whether the file is compressed flag size_t fsize; // file size time_t mtime; //The last modification time of the file time_t atime; //The last access time of the file std::string real_path; //The actual storage path of the file std::string pack_path; // Compressed package storage path std::string url; // File access URL bool NewBackupInfo(const std::string & amp;realpath) {<!-- --> FileUtil fu(realpath); if(fu.Exists() == false) {<!-- --> std::cout << "new backupinfo: file not exit" << std::endl; return false; } Config* config = Config::GetInstance(); std::string packdir = config->GetPackDir(); std::string packsuffix = config->GetPackFileSuffix(); std::string download_prefix = config->GetDownloadPrefix(); pack_flag = false; fsize = fu.FileSize(); mtime = fu.LastMtime(); atime = fu.LastATime(); real_path = realpath; pack_path = packdir + fu.FileName() + packsuffix; url = download_prefix + fu.FileName(); return true; } }BackupInfo; classDataManager {<!-- --> public: DataManager() {<!-- --> _backup_file = Config::GetInstance()->GetBackupFile(); // Get the file for persistent storage of backup information pthread_rwlock_init( & amp;_rwlock, NULL); // Initialize read-write lock InitLoad(); // Initial loading } ~DataManager() {<!-- --> pthread_rwlock_destroy( & amp;_rwlock); // Release the read-write lock } bool Insert(const BackupInfo & amp;info) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); _table[info.url] = info; pthread_rwlock_unlock( & amp;_rwlock); return true; } bool Updata(const BackupInfo & amp;info) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); _table[info.url] = info; pthread_rwlock_unlock( & amp;_rwlock); return true; } bool GetOneByURL(const std::string & amp;url, BackupInfo* info) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); auto it = _table.find(url); if(it == _table.end()) {<!-- --> pthread_rwlock_unlock( & amp;_rwlock); return false; } *info = it->second; pthread_rwlock_unlock( & amp;_rwlock); return true; } bool GetOneByRealpath(const std::string & realpath, BackupInfo* info) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); auto it = _table.begin(); for(; it != _table.end(); + + it) {<!-- --> if(it->second.real_path == realpath) {<!-- --> *info = it->second; pthread_rwlock_unlock( & amp;_rwlock); return true; } } pthread_rwlock_unlock( & amp;_rwlock); return false; } bool GetAll(std::vector<BackupInfo>* array) {<!-- --> pthread_rwlock_wrlock( & amp;_rwlock); auto it = _table.begin(); for(; it != _table.end(); + + it) {<!-- --> array->push_back(it->second); } pthread_rwlock_unlock( & amp;_rwlock); return true; } boolStorage() {<!-- --> // 1. Get all data std::vector<BackupInfo> array; GetAll(&array); // 2. Add to Json::Value Json::Value root; for(int i = 0; i < array.size(); i + + ) {<!-- --> Json::Value item; root["pack_flag"] = array[i].pack_flag; root["fsize"] = (Json::Int64)array[i].fsize; root["atime"] = (Json::Int64)array[i].atime; root["mtime"] = (Json::Int64)array[i].mtime; root["real_path"] = array[i].real_path; root["pack_path"] = array[i].pack_path; root["url"] = array[i].url; root.append(item); //Add array elements } // 3. Serialize Json::Value std::string body; JsonUtil::Serialize(root, & amp;body); // 4. Write file FileUtil fu(_backup_file); fu.SetContent(body); return true; } boolInitLoad() {<!-- --> // 1. Read the data from the file FileUtil fu(_backup_file); if(fu.Exists() == false) {<!-- --> return true; } std::string body; fu.GetContent( & amp;body); // 2. Deserialization Json::Value root; JsonUtil::Unserialize(body, & amp;root); // 3. Add the data in Json::Value obtained by deserialization to the table for(int i = 0; i < root.size(); i + + ) {<!-- --> BackupInfo info; info.pack_flag = root[i]["pack_flag"].asBool(); info.fsize = root[i]["fsize"].asInt64(); info.atime = root[i]["atime"].asInt64(); info.mtime = root[i]["mtime"].asInt64(); info.pack_path = root[i]["pack_path"].asString(); info.real_path = root[i]["real_path"].asString(); info.url = root[i]["url"].asString(); Insert(info); } return true; } private: std::string _backup_file; // Persistent storage of backup information pthread_rwlock_t _rwlock; // read-write lock std::unordered_map<std::string, BackupInfo> _table; // Data information organization }; } #endif