Introduction
Under normal circumstances, if it is a multi-copy data store, it is easy to repair. Deleting one or two BE can also be restored according to Doris’s own recovery mechanism. However, sometimes some tables may not have multiple copies, so it is a bit confusing. .
Case description: The data of the master node of fe was accidentally deleted, causing the cluster to go down directly. If it is a single fe, you can use the snapshot function of cloud storage to restore to the specified time point. Metadata storageIf the fe of multiple nodes deletes the master, you can use the following operations.
Recover Fe’s data
Question 1
(Remember to make a backup copy before operation), delete the files in the execution directory, such as ROLE and VERSION in the following directory (if it can start normally, there is no need to delete it, if it cannot start, then delete ROLE and VERSION):
/apache-doris-fe-1.2.0-bin-x86_64/doris-meta/image
Delete ROLE and VERSION, do not delete the image file (if it can start normally, then there is no need to delete it, if it cannot start, then delete ROLE and VERSION)
Then configure fe.conf as follows (when you restore to the point where you can connect to query metadata, stop the service and cancel the following configuration. After recovery, change the cluster id of the newly generated VERSION data to the previous cluster id. The reason is It must be the same as Be’s cluster ID)
metadata_failure_recovery=true
Question 2
The following problem is due to a problem with fe data synchronization. If there are multiple fe, delete this node and rejoin the node. If not, use the above method to delete ROLE and VERSION in Fe metadata to restore.
#Execute on sql command line ALTER SYSTEM DROP FOLLOWER "100.200.0.36:9010"; ALTER SYSTEM ADD FOLLOWER "100.200.0.36:9010"; #The node to be added, 10.200.8.117 is the master node. The port of the node to be added must be the same as all the ports in the cluster. ./bin/start_fe.sh --helper 100.200.8.117:9010 --daemon
Start Be
After restoring Fe’s data, you can start Be (if Be uses supervisord, it will be closed first. If it is not closed, it will have a LOCK file in the meta directory and stream_load of Be’s data storage directory. This is because After enabling auto-start, if you want to debug the lock contention problem later, just delete the LOCK file and restart.)
Fix the problem of deleting form copies
Some Be’s sst files may be lost, so you can use the cloud’s disk snapshot function to restore data to a specified time. The following problems will occur after recovery.
Problem 1: The corresponding tablet cannot be found (fill it with a blank tablet)
You can use the following command to overwrite a blank tablet, then nearly one day’s worth of data will be lost (see the doris official website for details).
ADMIN SET FRONTEND CONFIG ("recover_with_empty_tablet" = "true");
After repairing, change it to false
ADMIN SET FRONTEND CONFIG ("recover_with_empty_tablet" = "false");
Question 2: The Fe data version is higher than the Be data version
ERROR 1105 (HY000): errCode = 2, detailMessage = (100.200.8.186)[INTERNAL_ERROR] failed to initialize storage reader. tablet=6708126.992547307.aa48977d2d69366b-180d241750a968ae, res=[INTE RNAL_ERROR]fail to find path in version_graph. spec_version : 0-132, backend=100.200.8.186
Repair steps
================================================ ============================= #Query the error table mysql> select * from databasename.tablename limit 10; ERROR 1105 (HY000): errCode = 2, detailMessage = (100.20.8.16)[INTERNAL_ERROR] failed to initialize storage reader. tablet=20289334.1750657277.ab4aa3a697bb5bec-ea5fe4f921c9f8ba, res=[INTERNAL_ERROR]f ail to find path in version_graph. spec_version: 0- 48974, backend=100.20.8.16 #Query the detailed information of the tablet, and then execute the command below DetailCmd mysql> show tablet 20289334; + ------------------------------------ + ------------ ---- + ---------------- + ---------------- + -------- + -- ------- + ------------- + --------- + -------- + ------- + - -------------------------------------------------- -------------------- + | DbName | TableName | PartitionName | IndexName | DbId | TableId | PartitionId | IndexId | IsSync | Order | DetailCmd | + ------------------------------------ + ------------ ---- + ---------------- + ---------------- + -------- + -- ------- + ------------- + --------- + -------- + ------- + - -------------------------------------------------- -------------------- + | default_cluster:databasename | tablename | tablename | tablename | 993914 | 6091980 | 20289269 | 6091981 | true | 16 | SHOW PROC '/dbs/993914/6091980/partitions/20289269/6091981/20289334'; | + ------------------------------------ + ------------ ---- + ---------------- + ---------------- + -------- + -- ------- + ------------- + --------- + -------- + ------- + - -------------------------------------------------- -------------------- + 1 row in set (0.00 sec) mysql> SHOW PROC '/dbs/993914/6091980/partitions/20289269/6091981/20289334'; + ---------- + ---------- + ---------- + --------------- ---- + ------------------ + --------------- + ---------- -- + --------------- + ---------------- + ---------- + --- ----- + ------- + -------------- + -------------------------- - +------------------------------------------------ --- + -------------------------------------------------- ------------------ + | ReplicaId | BackendId | Version | LstSuccessVersion | LstFailedVersion | LstFailedTime | SchemaHash | LocalDataSize | RemoteDataSize | RowCount | State | IsBad | VersionCount | PathHash | MetaUrl | CompactionStatus | + ---------- + ---------- + ---------- + --------------- ---- + ------------------ + --------------- + ---------- -- + --------------- + ---------------- + ---------- + --- ----- + ------- + -------------- + -------------------------- - +------------------------------------------------ --- + -------------------------------------------------- ------------------ + | 20289335 | 992447 | 48974 | 48974 This is the final successful version, that is, the data version that Fe has but Be does not! ! ! | -1 | NULL | 1750657277 | 23244725 | 0 | 758834 | NORMAL | false | 9 | -1695231754850110287 | http://10.240.8.73:8040/api/meta/header/20289334 | 0 /api/compaction/show?tablet_id=20289334 | | 20289337 | 992604 | 48974 | 48974 | -1 | NULL | 1750657277 | 23244725 | 0 | 758834 | NORMAL | false | 9 | -7776024449396720965 | ta/header/20289334 | http ://10.240.8.9:8040/api/compaction/show?tablet_id=20289334 | | 29024516 | 993073 | 48974 | 48974 | -1 | NULL | 1750657277 | 23220943 | 0 | 758485 | NORMAL | false | 7 | -7256810246008465985 | http://100.20.8.16:8040/api/ meta/header/20289334 | http ://100.20.8.16:8040/api/compaction/show?tablet_id=20289334 | + ---------- + ---------- + ---------- + --------------- ---- + ------------------ + --------------- + ---------- -- + --------------- + ---------------- + ---------- + --- ----- + ------- + -------------- + -------------------------- - +------------------------------------------------ --- + -------------------------------------------------- ------------------ + #Check the merged versions. Since the completed version is 48974, the following are the successfully merged versions [48410-48665], then start merging from 48666 until 48974 (here is to fill in the data that Be does not have but Fe has) curl http://100.20.8.16:8040/api/compaction/show?tablet_id=20289334 [root@doris1 ~]# curl http://100.20.8.16:8040/api/compaction/show?tablet_id=20289334 { "cumulative policy type": "SIZE_BASED", "cumulative point": 2, "last cumulative failure time": "2023-10-19 18:57:10.751", "last base failure time": "1970-01-01 08:00:00.000", "last cumulative success time": "2023-10-19 18:57:10.751", "last base success time": "2023-10-19 18:57:10.751", "rowsets": [ "[0-1] 0 DATA NONOVERLAPPING 0200000000b943b1624293f95b56fef5e5a0a7526615fbb0 0", "[2-8892] 1 DATA NONOVERLAPPING 0200000000b943b2624293f95b56fef5e5a0a7526615fbb0 16.68 MB", "[8893-44310] 1 DATA NONOVERLAPPING 0200000000fd69e8624293f95b56fef5e5a0a7526615fbb0 4.62 MB", "[44311-44940] 1 DATA NONOVERLAPPING 020000000102d9dd624293f95b56fef5e5a0a7526615fbb0 531.21 KB", "[44941-47519] 1 DATA NONOVERLAPPING 020000000118285f624293f95b56fef5e5a0a7526615fbb0 252.91 KB", "[47520-48409] 1 DATA NONOVERLAPPING 02000000011fd182624293f95b56fef5e5a0a7526615fbb0 64.41 KB", "[48410-48665] 1 DATA NONOVERLAPPING 020000000121faaf624293f95b56fef5e5a0a7526615fbb0 21.43 KB" ], "missing_rowsets": [], "stale_rowsets": [], "stale version path": [] } #Check the merged versions. Since the completed version is 48974, the following are the successfully merged versions [48410-48665], so start merging from 48666 until 48974 curl -X POST "http://100.20.8.16:8040/api/pad_rowset?tablet_id=20289334 & amp;start_version=48666 & amp;end_version=48974"
Database deletion recovery
#Restore database RECOVER DATABASE example_db; #Restore table recover table table name; #Force deletion drop xxx force; #View recycle bin show trash; #Clear the recycle bin data of the specified be ADMIN CLEAN TRASH ON("be_ip:9050");
Prevent database deletion
1. First, the number of machine nodes needs to be >= the number of replicas.
2. Modify the number of historical partition copies: alter table modify partition(*) set (“replication_num” = “3”);
3. Dynamic partitioning modifies the number of future partition copies: ALTER TABLE example_db.mysql_table SET (“dynamic_partition.replication_num” = “3”);
4. Non-partitioned table: ALTER TABLE example_db.mysql_table SET (“replication_num” = “3”);
ADMIN SHOW REPLICA STATUS FROM database.table name; show partitions from database.table name;
Note: First operate the test table to see if the status of the test table is as expected. Run show partitions from table and show create table to check. show partitions from is the actual number of copies.
The knowledge points of the article match the official knowledge files, and you can further learn related knowledge. MySQL entry skill treeUsing database Create and delete database 76329 people are learning the system