Tag: dfs

Seaweedfs Erasure-coding in-depth analysis erasure coding Reed-Solomon code distributed object storage redundant error correction high availability test

Seaweedfs Erasure-coding Introduction https://github.com/seaweedfs/seaweedfs/wiki/Erasure-coding-for-warm-storage SeaweedFS implements RS(10,4), which allows 4 blocks to be lost out of every 10 hard disks and can still be accessed normally. It saves 3.6 times the disk space compared to copying data 5 times to achieve the same robustness. Service startup Start the master server, 4 volume servers, and a […]

FastDFS+Nginx – Build a file server locally and achieve remote access “port mapping” from outside

Link: https://www.linkinstars.com Foreword FastDFS is an open source lightweight distributed file system that manages files. Its functions include: file storage, file synchronization, file access (file upload, file download), etc., solving the problems of large-capacity storage and load balancing. It is especially suitable for online services based on files, such as photo album websites, video websites, […]

ETL toolDatax-ETL-SqlServerToHDFS

Personal homepage–Personal homepage ? ? Thanks for the likes and attention, we will make a little progress every day! come on! ? Table of Contents Personal homepage–Personal homepage ? 1. Overview of DataX 1.1 Introduction to DataX 1.2 DataX framework 1.3 Functional limitations 1.4 Support Data Channels 2. Configuration example 2.1 Environmental information 2.2 SQLServer […]

Solving ImportError: HDFStore requires PyTables, “No module named tables“ problem importing

Table of Contents Solving ImportError: HDFStore requires PyTables, “No module named ‘tables’” problem importing Step 1: Install PyTables library Step 2: Confirm that the PyTables library has been installed correctly Step 3: Rerun the program summary Introduction to PyTables library Main features of PyTables PyTables installation Solve ImportError: HDFStore requires PyTables, “No module named ‘tables’” […]

DFS, BFS of graphs, maze problems, search and backtracking template questions – full arrangement, Dijkstra

graph dfs-stack #include<iostream> using namespace std; #include<stack> int matrix[7][7] = { //A,B,C,D,E,F,G /*A*/ {0,0,1,1,0,1,0}, /*B*/ {0,0,1,0,0,0,0}, /*C*/ {1,1,0,1,0,0,0}, /*D*/ {1,0,1,0,0,0,0}, /*E*/ {0,0,0,0,0,0,1}, /*F*/ {1,0,0,0,0,0,1}, /*G*/ {0,0,0,0,1,1,0} }; bool vis[7];//mark array //Depth first search void dfs() { stack<int> s; //Start searching from A s.push(0); vis[0] = 1; while (!s.empty()) { int tmp = s.top(); s.pop(); cout […]

HDFS cluster NameNode high availability transformation

Article directory background High availability transformation Program implementation Environmental preparation Configuration file modification Application configuration Cluster status verification High availability verification Background Assume that there are currently three ZooKeeper servers, namely zk-01/02/03, and several DataNode servers; Currently, the Namenode of the HDFS cluster does not have high availability configuration. The Namenode and Secondary Namenode are […]

springboot hdfs operation tool class

Import dependencies <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.7.3</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.7.3</version> </dependency> Download winutils.exe and configure environment variables GitHub – srccodes/hadoop-common-2.2.0-bin: hadoop-common-2.2.0/bin HADOOP_HOME: The directory where the file is located\hadoop-common-2.2.0-bin-master\bin Add hdfs configuration information hdfs: path: hdfs://localhost:9000 username: hadoop01 Write code package com.iproinfo.refinedanalysis.service; import org.apache.hadoop.fs.BlockLocation; import org.springframework.web.multipart.MultipartFile; import java.util.List; import java.util.Map; public interface HdfsService { […]

Graph Theory 03-[Unweighted and Undirected]-Depth-first DFS traversal of graphs-path problem/detection cycle/bipartite graph

Article directory 1. Code warehouse 2. Single source path 2.1 Ideas 2.2 Main code 3. All point-pair paths 3.1 Ideas 3.2 Main code 4. Optimization of path problem-end recursion early 4.1 Ideas 4.2 Main code 5. Detection loop 5.1 Ideas 5.2 Main code 6. Bipartite graph 6.1 Ideas 6.2 Main code 6.2.1 Traverse each connected […]

Graph traversal-depth-first (DFS) traversal

Table of Contents Preface Depth-first traversal (DFS) 1. Basic concepts 2.Algorithmic thinking 3. Depth-first traversal of binary trees (example) Depth first traversal of graph 1. Depth-first traversal of graph adjacency matrix Idea analysis Code 2. Depth-first traversal of graph adjacency list Idea analysis Code recursive code non-recursive code 3. Comparison between adjacency matrix and adjacency […]