C++ project – cloud backup-⑤-Design and implementation of server-side data management module

Article directory

  • Column introduction
  • 1. What data needs to be managed?
  • 2. How to manage data
  • 3. Design and implementation of data information structure
  • 4.Data management design
  • 5. Data management class implementation
  • 6. Data management module organization

Introduction to the column

About the author: Hua Xiangyun, undergraduate student, C /C++ Rising Star Creator, Rising Star Program Mentor, Alibaba Cloud Expert Blogger, CSDN Content Partner… Dedicated to C/C++ and Linux learning.

Column introduction: This article is included in theC++ project – cloud backup

Related column recommendations: C Language Beginner Series, C Language Advanced Series< /em>, C++ Series, Data Structures and Algorithms, Linux< /strong>
Project Gitee link: https://gitee.com/li-yuanjiu/cloud-backup

1. What data needs to be managed

  • The actual storage path of the file: When the customer wants to download a file, the data is read from this file to respond;
  • File is compressed flag: Determine whether the file has been compressed;
  • Compressed package storage path: If a file is a non-hotspot file, it will be compressed. This is the path where the compressed package is stored;
    • If the client wants to download a file, it needs to decompress it first and then read the decompressed file data;

There are also various types of file attribute information:

  • File size;
  • File last modified time;
  • File last access time;
  • Resource path path in file access URL;

2. How to manage data

  • For data information access: Use hash table to manage data in memory, use file access URL as key, The data information structure is val. Access is fast. File access URL is used as the key because the client browser always uses URL as the request when downloading files;

  • Persistent storage management: Use JSON serialization to save all data to files;

3. Design and implementation of data information structure

  • The struct BackupInfo structure contains various attribute information of the files to be managed;
  • NewBackUpInfo is responsible for organizing the attribute information of the passed in file;
typedef struct BackupInfo
{<!-- -->
    bool pack_flag; // Whether the file is compressed flag
    size_t fsize; // file size
    time_t mtime; //The last modification time of the file
    time_t atime; //The last access time of the file
    std::string real_path; //The actual storage path of the file
    std::string pack_path; // Compressed package storage path
    std::string url; // File access URL
    bool NewBackupInfo(const std::string & amp;realpath)
    {<!-- -->
        FileUtil fu(realpath);
        if(fu.Exists() == false)
        {<!-- -->
            std::cout << "new backupinfo: file not exit" << std::endl;
            return false;
        }
        Config* config = Config::GetInstance();
        std::string packdir = config->GetPackDir();
        std::string packsuffix = config->GetPackFileSuffix();
        std::string download_prefix = config->GetDownloadPrefix();
        pack_flag = false;
        fsize = fu.FileSize();
        mtime = fu.LastMtime();
        atime = fu.LastATime();
        real_path = realpath;
        pack_path = packdir + fu.FileName() + packsuffix;
        url = download_prefix + fu.FileName();
        return true;
    }
}BackupInfo;

4. Data management design

The data management class is responsible for managing data information.

Data management DataManger includes the following members:

class DataManager
    {<!-- -->
    public:
        DataManager();
        ~DataManager();
        bool Insert(const BackupInfo & amp;info); // New
        bool Updata(const BackupInfo & amp;info); // Update
        bool GetOneByURL(const std::string & amp;url, BackupInfo* info); // Get file information through url
        bool GetOneByRealpath(const std::string & amp;realpath, BackupInfo* info); // Get file information through the actual path
        bool GetAll(std::vector<BackupInfo>* array); // Get all file information
        bool Storage(); // Every time data is updated or added, it must be stored persistently
        bool InitLoad(); // Initial loading, previous data must be loaded before each system startup
    private:
        std::string _backup_file; // Persistent storage of backup information
        pthread_rwlock_t _rwlock; // read-write lock
        std::unordered_map<std::string, BackupInfo> _table // Data information organization;
    };

5. Data management class implementation

class DataManager
{<!-- -->
public:
    DataManager()
    {<!-- -->
        _backup_file = Config::GetInstance()->GetBackupFile(); // Get the file for persistent storage of backup information
        pthread_rwlock_init( & amp;_rwlock, NULL); // Initialize read-write lock
        InitLoad(); // Initial loading
    }
    ~DataManager()
    {<!-- -->
        pthread_rwlock_destroy( & amp;_rwlock); // Release the read-write lock
    }
    bool Insert(const BackupInfo & amp;info)
    {<!-- -->
        pthread_rwlock_wrlock( & amp;_rwlock);
        _table[info.url] = info;
        pthread_rwlock_unlock( & amp;_rwlock);
        return true;
    }
    bool Updata(const BackupInfo & amp;info)
    {<!-- -->
        pthread_rwlock_wrlock( & amp;_rwlock);
        _table[info.url] = info;
        pthread_rwlock_unlock( & amp;_rwlock);
        return true;
    }
    bool GetOneByURL(const std::string & amp;url, BackupInfo* info)
    {<!-- -->
        pthread_rwlock_wrlock( & amp;_rwlock);
        auto it = _table.find(url);
        if(it == _table.end())
        {<!-- -->
            pthread_rwlock_unlock( & amp;_rwlock);
            return false;
        }
        *info = it->second;
        pthread_rwlock_unlock( & amp;_rwlock);
        return true;
    }
    bool GetOneByRealpath(const std::string & realpath, BackupInfo* info)
    {<!-- -->
        pthread_rwlock_wrlock( & amp;_rwlock);
        auto it = _table.begin();
        for(; it != _table.end(); + + it)
        {<!-- -->
            if(it->second.real_path == realpath)
            {<!-- -->
                *info = it->second;
                pthread_rwlock_unlock( & amp;_rwlock);
                return true;
            }
        }
        pthread_rwlock_unlock( & amp;_rwlock);
        return false;
    }
    bool GetAll(std::vector<BackupInfo>* array)
    {<!-- -->
        pthread_rwlock_wrlock( & amp;_rwlock);
        auto it = _table.begin();
        for(; it != _table.end(); + + it)
        {<!-- -->
            array->push_back(it->second);
        }
        pthread_rwlock_unlock( & amp;_rwlock);
        return true;
    }
    boolStorage()
    {<!-- -->
        // 1. Get all data
        std::vector<BackupInfo> array;
        GetAll(&array);
        // 2. Add to Json::Value
        Json::Value root;
        for(int i = 0; i < array.size(); i + + )
        {<!-- -->
            Json::Value item;
            root["pack_flag"] = array[i].pack_flag;
            root["fsize"] = (Json::Int64)array[i].fsize;
            root["atime"] = (Json::Int64)array[i].atime;
            root["mtime"] = (Json::Int64)array[i].mtime;
            root["real_path"] = array[i].real_path;
            root["pack_path"] = array[i].pack_path;
            root["url"] = array[i].url;
            root.append(item); //Add array elements
        }
        // 3. Serialize Json::Value
        std::string body;
        JsonUtil::Serialize(root, & amp;body);
        // 4. Write file
        FileUtil fu(_backup_file);
        fu.SetContent(body);
        return true;
    }
    boolInitLoad()
    {<!-- -->
        // 1. Read the data from the file
        FileUtil fu(_backup_file);
        if(fu.Exists() == false)
        {<!-- -->
            return true;
        }
        std::string body;
        fu.GetContent( & amp;body);
        // 2. Deserialization
        Json::Value root;
        JsonUtil::Unserialize(body, & amp;root);
        // 3. Add the data in Json::Value obtained by deserialization to the table
        for(int i = 0; i < root.size(); i + + )
        {<!-- -->
            BackupInfo info;
            info.pack_flag = root[i]["pack_flag"].asBool();
            info.fsize = root[i]["fsize"].asInt64();
            info.atime = root[i]["atime"].asInt64();
            info.mtime = root[i]["mtime"].asInt64();
            info.pack_path = root[i]["pack_path"].asString();
            info.real_path = root[i]["real_path"].asString();
            info.url = root[i]["url"].asString();
            Insert(info);
        }
        return true;
    }
private:
    std::string _backup_file; // Persistent storage of backup information
    pthread_rwlock_t _rwlock; // read-write lock
    std::unordered_map<std::string, BackupInfo> _table; // Data information organization
};

6. Data management module organization

We put the implementations of BackpInfo and DataManger into data.hpp.

// data.hpp
#ifndef __MY_DATA__
#define __MY_DATA__
#include <unordered_map>
#include <pthread.h>
#include "util.hpp"
#include "config.hpp"

namespace cloud
{<!-- -->
    typedef struct BackupInfo
    {<!-- -->
        bool pack_flag; // Whether the file is compressed flag
        size_t fsize; // file size
        time_t mtime; //The last modification time of the file
        time_t atime; //The last access time of the file
        std::string real_path; //The actual storage path of the file
        std::string pack_path; // Compressed package storage path
        std::string url; // File access URL
        bool NewBackupInfo(const std::string & amp;realpath)
        {<!-- -->
            FileUtil fu(realpath);
            if(fu.Exists() == false)
            {<!-- -->
                std::cout << "new backupinfo: file not exit" << std::endl;
                return false;
            }
            Config* config = Config::GetInstance();
            std::string packdir = config->GetPackDir();
            std::string packsuffix = config->GetPackFileSuffix();
            std::string download_prefix = config->GetDownloadPrefix();
            pack_flag = false;
            fsize = fu.FileSize();
            mtime = fu.LastMtime();
            atime = fu.LastATime();
            real_path = realpath;
            pack_path = packdir + fu.FileName() + packsuffix;
            url = download_prefix + fu.FileName();
            return true;
        }
    }BackupInfo;
    classDataManager
    {<!-- -->
    public:
        DataManager()
        {<!-- -->
            _backup_file = Config::GetInstance()->GetBackupFile(); // Get the file for persistent storage of backup information
            pthread_rwlock_init( & amp;_rwlock, NULL); // Initialize read-write lock
            InitLoad(); // Initial loading
        }
        ~DataManager()
        {<!-- -->
            pthread_rwlock_destroy( & amp;_rwlock); // Release the read-write lock
        }
        bool Insert(const BackupInfo & amp;info)
        {<!-- -->
            pthread_rwlock_wrlock( & amp;_rwlock);
            _table[info.url] = info;
            pthread_rwlock_unlock( & amp;_rwlock);
            return true;
        }
        bool Updata(const BackupInfo & amp;info)
        {<!-- -->
            pthread_rwlock_wrlock( & amp;_rwlock);
            _table[info.url] = info;
            pthread_rwlock_unlock( & amp;_rwlock);
            return true;
        }
        bool GetOneByURL(const std::string & amp;url, BackupInfo* info)
        {<!-- -->
            pthread_rwlock_wrlock( & amp;_rwlock);
            auto it = _table.find(url);
            if(it == _table.end())
            {<!-- -->
                pthread_rwlock_unlock( & amp;_rwlock);
                return false;
            }
            *info = it->second;
            pthread_rwlock_unlock( & amp;_rwlock);
            return true;
        }
        bool GetOneByRealpath(const std::string & realpath, BackupInfo* info)
        {<!-- -->
            pthread_rwlock_wrlock( & amp;_rwlock);
            auto it = _table.begin();
            for(; it != _table.end(); + + it)
            {<!-- -->
                if(it->second.real_path == realpath)
                {<!-- -->
                    *info = it->second;
                    pthread_rwlock_unlock( & amp;_rwlock);
                    return true;
                }
            }
            pthread_rwlock_unlock( & amp;_rwlock);
            return false;
        }
        bool GetAll(std::vector<BackupInfo>* array)
        {<!-- -->
            pthread_rwlock_wrlock( & amp;_rwlock);
            auto it = _table.begin();
            for(; it != _table.end(); + + it)
            {<!-- -->
                array->push_back(it->second);
            }
            pthread_rwlock_unlock( & amp;_rwlock);
            return true;
        }
        boolStorage()
        {<!-- -->
            // 1. Get all data
            std::vector<BackupInfo> array;
            GetAll(&array);
            // 2. Add to Json::Value
            Json::Value root;
            for(int i = 0; i < array.size(); i + + )
            {<!-- -->
                Json::Value item;
                root["pack_flag"] = array[i].pack_flag;
                root["fsize"] = (Json::Int64)array[i].fsize;
                root["atime"] = (Json::Int64)array[i].atime;
                root["mtime"] = (Json::Int64)array[i].mtime;
                root["real_path"] = array[i].real_path;
                root["pack_path"] = array[i].pack_path;
                root["url"] = array[i].url;
                root.append(item); //Add array elements
            }
            // 3. Serialize Json::Value
            std::string body;
            JsonUtil::Serialize(root, & amp;body);
            // 4. Write file
            FileUtil fu(_backup_file);
            fu.SetContent(body);
            return true;
        }
        boolInitLoad()
        {<!-- -->
            // 1. Read the data from the file
            FileUtil fu(_backup_file);
            if(fu.Exists() == false)
            {<!-- -->
                return true;
            }
            std::string body;
            fu.GetContent( & amp;body);
            // 2. Deserialization
            Json::Value root;
            JsonUtil::Unserialize(body, & amp;root);
            // 3. Add the data in Json::Value obtained by deserialization to the table
            for(int i = 0; i < root.size(); i + + )
            {<!-- -->
                BackupInfo info;
                info.pack_flag = root[i]["pack_flag"].asBool();
                info.fsize = root[i]["fsize"].asInt64();
                info.atime = root[i]["atime"].asInt64();
                info.mtime = root[i]["mtime"].asInt64();
                info.pack_path = root[i]["pack_path"].asString();
                info.real_path = root[i]["real_path"].asString();
                info.url = root[i]["url"].asString();
                Insert(info);
            }
            return true;
        }
    private:
        std::string _backup_file; // Persistent storage of backup information
        pthread_rwlock_t _rwlock; // read-write lock
        std::unordered_map<std::string, BackupInfo> _table; // Data information organization
    };
}
#endif

syntaxbug.com © 2021 All Rights Reserved.