1) Elasticsearch snapshot backup
Advantages: Take a snapshot through snapshot, and then define a snapshot backup strategy, which can realize automatic storage of snapshots. You can define various strategies to meet your different backup needs.
Disadvantages: Restoration is not flexible enough. Taking snapshots for backup is very fast, but there is no way to restore at will, similar to virtual machine snapshots.
1. Configure backup directory
In the configuration file of elasticsearch.yml, indicate path.repo which can be used as the backup path, as shown below:
path.repo: ["/mount/backups", "/mount/longterm_backups"]
After configuration, you can use the snapshot api to create a repository. As follows, we create a repository named my_backup.
PUT /_snapshot/my_backup {<!-- --> "type": "fs", "settings": {<!-- --> "location": "/mount/ backups/my_backup" } }
2. Start backup through API interface
With repostiroy, we can make backups, also called snapshots, which record the current status of the data. As shown below we create a snapshot named snapshot_1.
PUT /_snapshot/my_backup/snapshot_1?wait_for_completion=true
[Warm reminder] If wait_for_completion is true, it means that the api will return the result after the backup execution is completed. Otherwise, it will be executed asynchronously by default. In order to see the effect immediately, we set this parameter here. There is no need to set this parameter when executing online. Just let it execute asynchronously in the background.
3. Incremental backup
PUT /_snapshot/my_backup/snapshot_2?wait_for_completion=true
When the execution is completed, you will find that the size of /mount/backups/my_backup has become larger. This shows that new data backup has come in. One thing to note is that when you take multiple snapshots in the same repository, elasticsearch will check whether the data segment file to be backed up has changed. If there is no change, it will not be processed. Otherwise, only the changed segment file will be backed up. . This actually implements incremental backup.
4. Data recovery
The recovery function can be quickly implemented by calling the following API:
POST /_snapshot/my_backup/snapshot_1/_restore?wait_for_completion=true {<!-- --> "indices": "index_1", "rename_replacement": "restored_index_1" }
2) elasticdump backup and migration es data
Export index data to file (backup)
# Export index Mapping data elasticdump \ --input=http://es instance IP:9200/index_name/index_type \ --output=/data/my_index_mapping.json \ # Storage directory --type=mapping # Export index Data elasticdump \ --input=http://es instance IP:9200/index_name/index_type \ --output=/data/my_index.json \ --type=data
Importing index data files into index (restore)
# Import Mapping data into index elasticdump \ --output=http://es instance IP:9200/index_name \ --input=/home/indexdata/roll_vote_mapping.json \ # Import data directory --type=mapping # Import ES document data into the index elasticdump \ --output=http:///es instance IP:9200/index_name \ --input=/home/indexdata/roll_vote.json \ --type=data
Backup data can be directly imported into another es cluster
elasticdump --input=http://127.0.0.1:9200/test_event --output=http://127.0.0.2:9200/test_event --type=data
type type
type is the ES data export and import type. The Elasticdump tool supports the following data types:
type type description
Index mapping structure data of mapping ES
data ES data
settings The default configuration of the index library of ES
analyzer ES tokenizer
Template structure data of template ES
alias index alias for ES
3) esm backup and migration es data
Back up es data
esm -s http://10.33.8.103:9201 -x "petition_data" -b 5 --count=5000 --sliced_scroll_size=10 --refresh -o=./es_backup.bin
-w indicates the number of threads
-b indicates the data size of a bulk request, the unit is MB, the default is 5M
-c Number of scroll requests at one time
Import and restore es data
esm -d http://172.16.20.20:9201 -y "petition_data6" -c 5000 -b 5 --refresh -i=./dump.bin
https://zhuanlan.zhihu.com/p/540189644
https://www.cnblogs.com/windysai/p/14400830.html
#!/bin/bash
#db_backups_conf.txt file path
db_backups_conf="/home/ljy/scripts/elasticsearch_backup_config.txt"
base_path='/backup/elkdata'
mv_elk_bath='/backup/elkmvdata'
#This machine only keeps snapshots of the last 3 days
find ${mv_elk_bath}/ -type f -mtime + 2 -exec rm -rf {<!-- -->} \; > /dev/null 2> & amp;1
## Move the original one first
# Get the previous day’s: year, month, day
y=`date + "%Y" -d "-1day"`
ym=`date + "%Y%m" -d "-1day"`
d=`date " + %d" -d "-1day"`
if [ ! -d ${mv_elk_bath}/${y}/${ym}/${d} ];
then
mkdir -p ${mv_elk_bath}/${y}/${ym}/${d}
scp -rp ${base_path}/* ${mv_elk_bath}/${y}/${ym}/${d}/
fi
#Determine whether the configuration information file exists
if [ -f "${db_backups_conf}" ];then
echo $(date + '%Y-%m-%d %H:%M:%S')" Found that the backup file configuration information file exists"
#Get the content before the equal sign as the Key value in the map
dbArrOne=($(awk -F'[=]' '{print $1}' ${<!-- -->db_backups_conf} ))
#Get the content after the equal sign as the value in the map
dbArrTwo=($(awk -F'[=]' '{print $2}' ${<!-- -->db_backups_conf}))
#Create an empty map
declare -A map=()
#Store the information in the db_backups_conf configuration file in the map through looping
for((i=0;i<${#dbArrOne[@]};i + + ))
do
map[${dbArrOne[i]}]=${dbArrTwo[i]}
done
#Get the string of the index name to be backed up
indexNames=${map["indexNames"]}
#Get the default string delimiter
old_ifs="$IFS"
#Set the string delimiter to comma
IFS=","
#Separate the string of index name value to be backed up and obtain an array
indexNamesArr=($indexNames)
#Reset the string delimiter to the default delimiter
IFS="$old_ifs"
#Get the current year, month and day
saveday=$(date + %Y%m%d)
#Get the year, month and day that exceed the number of backup days
delday=$(date -d ${<!-- -->map["backupsFileDay"]}' days ago' + %Y%m%d)
#Execute the command to back up es data
{<!-- -->
#Check whether the es access address is valid
esStatus=$(curl -s -m 5 -IL http://${<!-- -->map["ipAddress"]}:${<!-- -->map["portNumber\ "]}|grep 200)
if [ "$esStatus" != "" ];then
echo $(date + '%Y-%m-%d %H:%M:%S')" es address access is normal: http://"${map["ipAddress"]} ":"${map["portNumber"]}",Start backing up data"
#Traverse the backup index names, delete old backups one by one, and restart the full backup
for indexName in ${indexNamesArr[@]};
do
echo $indexName;
## Determine whether the index exists one by one
indexStatus=$(curl -s -m 5 -IL http://${<!-- -->map["ipAddress"]}:${<!-- -->map["portNumber\ "]}/${<!-- -->indexName}|grep 200)
if [ "$indexStatus" != "" ];then
#Backup first, then delete old backups
if [ ${map["backupBeforeDelete"]} == "yes" ];then
echo $(date + '%Y-%m-%d %H:%M:%S')" Start synchronizing backup index: "${indexName}_backup_${saveday}
curl -XPUT '127.0.0.1:9500/_snapshot/elk_backup/'${indexName}'_backup_'${saveday}'?wait_for_completion=true' -d '{"indices":\ "'${indexName}'"}'
echo $(date + '%Y-%m-%d %H:%M:%S')" Complete synchronization backup index: "${indexName}_backup_${saveday}
#Delete old backups
echo $(date + '%Y-%m-%d %H:%M:%S')" Start deleting backup index: "${indexName}_backup_${delday}
curl -XDELETE '127.0.0.1:9500/_snapshot/elk_backup/'${indexName}'_backup_'${delday}''
echo $(date + '%Y-%m-%d %H:%M:%S')" Complete deletion of backup index: "${indexName}_backup_${delday}
fi
else
echo $(date + '%Y-%m-%d %H:%M:%S')" Index cannot exist in es: "${indexName}
fi
done
fi
}
else
echo "File does not exist"
fi
Script to backup to NAS
Daily and monthly (only 1 backup per month)
#/bin/bash
#Date: 2020-08-02
#Script Name: elk_logs_bak
#Back up elk snapshot data to nas
# Log location
base_path='/backup/elkdata'
nas_bak='/nasdir/elk_bak/day_bak'
# Get the current year information and month information
ym=$(date + "%Y%m")
dy=`date " + %Y%m%d"`
d=`date " + %d"`
#Create folders by year and month
if [ ! -d ${nas_bak}/${ym}/${d} ];
then
sudo mkdir -p ${nas_bak}/${ym}/${d}
fi
scp -rp ${base_path}/* ${nas_bak}/${ym}/${d}/
#Back up the last day of data every month
nasmonth_bak='/nasdir/elk_bak/month_bak'
lastday=`date -d"$(date -d"1 month" + "%Y%m01") -1 day" + "%d"`
thisday=`date + "%d"`
#Keep the last day of the month log
if [ $thisday == lastday ];
then
mkdir -p ${nasmonth_bak}/${ym}/
scp -rp ${nas_bak}/${ym}/${d}/* ${nasmonth_bak}/${ym}/
fi
find ${nas_bak}/${ym}/ -type f -mtime + 30 -exec rm -rf {<!-- -->} \; > /dev/null 2> & amp;1