grep
Introduction
Purpose
grep is a line filtering tool; used to filter lines based on keywords.
Grammar
grep [options] 'keyword' filename #Search using keywords in file names
Options
-i: case insensitive -v: Find lines that do not contain the specified content, reverse selection -w: search by word -o: Print matching keywords -c: Count the number of matches -n: display line number -r: Traverse the directory search layer by layer -A: Display the matching line and how many lines follow it -B: Display the matching line and how many lines before it -C: Display how many lines before and after the matching line -l: Only list matching file names -L: List unmatched file names -e: Use regular matching -E: Use extended regular matching ^keyWord: starts with keyword keyWord$: ends with keyword ^$: matches empty lines --color=auto: Add color to the found keyword part and display it
Help
man grep
Examples
Filter out the keywords in the file and display the keywords in color
- Normal search
grep --color=auto 'root' /tmp/passwd
- Alias settings
#Temporary settings (only for the current user) alias grep='grep --color=auto' #Permanent settings (for all users) echo "alias grep='grep --color=auto'" >> /etc/bashrc source /etc/bashrc
Filter out the lines containing keywords in the file and display the line number
grep -n 'root' /tmp/passwd #-n: number, indicating row
Filter out keywords in the file and ignore case matching
grep -i 'root' /tmp/passwd grep -i 'root' /tmp/passwd
Filter out the keywords in the file and display the keywords that start or end with the keyword
grep '^root' /tmp/passwd grep 'bash$' /tmp/passwd
Filter out the keywords in the file and display the lines that do not contain the keywords
grep -v 'root' /tmp/passwd
Filter out the keywords in the file and display the first few lines, the last few lines, or the upper and lower lines containing the keywords
grep -A 3 'ftp' /tmp/passwd grep -B 3 'ftp' /tmp/passwd grep -C 3 'ftp' /tmp/passwd #-A: after, after #-B: before, before #-C：
Filter out the keywords in the file and display the lines containing the keywords as words
grep -w 'word' /tmp/passwd #-w: word, word
Filter out the keywords in the file and only display the keywords themselves
grep -o 'root' /tmp/passwd
Filter out keywords in the file and count the number of times the keyword appears
grep -c 'root' /tmp/passwd
cut
Introduction
Purpose
cut is a column interception tool; used for column interception.
Grammar
cut option file name
Options
-c: Split and intercept in character units (the number following indicates the number of characters) -d: Custom delimiter, the default tab character is "\t" -f: Used together with -d to specify which area to intercept (the following number indicates the line number)
Help
man cut
Examples
Use “:” as the separator to intercept the content of the first column
cut -d: -f1 /tmp/passwd cut -d ":" -f1 /tmp/passwd
Use “:” as the separator to intercept the contents of the first and last columns of the first three rows
cut -d ":" -f1,7 /tmp/passwd | head -n 3 #"," means sum, -f1,7 means columns 1 and 7
Intercept the first to fifth characters in the file
cut -c1-5 /tmp/passwd #"-" means to, -c1-5 means the 1st to 5th characters
Intercept the content from the 10th character to the last character in the file
cut -c10- /tmp/passwd
sort
Introduction
Purpose
The sort tool is used for sorting; it takes each line of the file as a unit, compares them by ASCII code value from the first character backward, and finally outputs them in ascending order.
Grammar
sort option file name
Options
-u: remove duplicate lines -r: Sort in descending order, the default is ascending order -o: Output the sorting results to a file, similar to the redirection symbol -n: Sort numerically, the default is sorting by characters -t: delimiter -k: Column N -b: Ignore leading spaces -R: Random sorting, the results of each run are different
Help
man sort
Examples
Arrange in ascending order according to user UID
sort -n -t ":" -k 3 /tmp/passwd #-n: Sort numerically #-t: Specify delimiter #-k: Specify which column
Sort in descending order by user’s UID
sort -n -r -t ":" -k 3 /tmp/passwd #-n: Sort numerically #-t: Specify delimiter #-k: Specify which column #-r: Sort in descending order
Arrange numerically
sort -n /tmp/passwd
Sort by numbers and remove duplicates
sort -n -u /tmp/passwd
Arrange numerically and redirect results to a new file
sort -n -u /tmp/passwd -o /tmp/passwd.bak
uniq
Introduction
Purpose
uniq is used to remove consecutive duplicates
Grammar
uniq option filename
Options
-i: ignore case -c: Count the number of repeated lines -d: Show only duplicate lines
Help
man uniq
Examples
Remove consecutive duplicate rows and return the deduplicated results
uniq /tmp/passwd
Only show duplicate rows
uniq -d /tmp/passwd
Count the number of duplicate rows
uniq -c -d /tmp/passwd
tee
Introduction
Purpose
The tee tool reads from standard input and writes to standard output and files, that is, bidirectional override redirection (screen output | text input)
Grammar
tee option file name
Options
-a: Bidirectional append redirection
Help
man tee
Examples
Print a copy of the screen and overwrite the printed content into the file
echo "wang ming" | tee test.txt
Print a copy of the screen and append the printed content to the file
echo "wang ming qu" | tee -a test.txt
diff
Introduction
Purpose
The diff tool is used to compare differences in files line by line.
Note: diff describes the two files in different ways, telling us how to change the first file to match the second file.
Grammar
diff option filename1 filename2
Options
-b: Do not check spaces -B: Do not check for blank lines -i: Do not check case -w: ignore all spaces --normal: Comparison results, displayed in normal format (default is normal format) -c: Comparison results, displayed in context format -u: Compare results and display in merged format
Help
man diff
Examples
Compare the similarities and differences between two text files and display them normally
diff old.txt new.txt 1,3d0 #Delete lines 1 to 3 of the first file to match line 0 of the second file, d means delete < wang ming qu *5 #The less than sign "<" indicates the file on the left, that is, the old.txt file <wangmingqu*5 <wangmingqu*5 6c3,4 #The 6th line of the first file must be modified to match the 3rd to 4th lines of the second file, c means modification < wang ming --- # represents the delimiter > wang ming qu > wang ming qu 13a12,15 #Add line 13 of the first file to match lines 12 to 15 of the second file, a means add > wang quan zi zhi #The greater than sign ">" indicates the file on the right, that is, the new.txt file > wang quan zi zhi > wang quan zi zhi > wang quan zi zhi
Compare the similarities and differences between two text files and display them in context
diff -c old.txt new.txt *** old.txt 2023-10-26 01:32:09.085672356 + 0800 #Definition *** represents the file old.txt --- new.txt 2023-10-26 01:42:15.243662950 + 0800 #Definition --- represents the file new.txt *************** # represents the delimiter *** 1,9 **** # Lines 1 to 9 of the first file - wang ming qu *5 #It needs to be deleted before it can match the second file, - means delete -wangmingqu*5 -wangmingqu*5 wang ming qu wang ming qu ! wang ming #needs modification to match the second file, ! means modification wang ming wang ming wang ming --- 1,7 ---- #Lines 1 to 7 of the second file wang ming qu wang ming qu ! wang ming qu #The first file needs to be modified to "wang ming qu" to match the second file, ! means modification ! wang ming qu wang ming wang ming wang ming *************** ***11,13**** --- 9,15 ---- wang wang wang + wang quan zi zhi #The first file needs to be added as "wang quan zi zhi" to match the second file, + means adding + wang quan zi zhi + wang quan zi zhi + wang quan zi zhi
Compare the similarities and differences between two text files and display them together
diff -u old.txt new.txt --- old.txt 2023-10-26 01:32:09.085672356 + 0800 #Definition --- represents the old.txt file + + + new.txt 2023-10-26 01:42:15.243662950 + 0800 #Definition + + + represents new.txt file @@ -1,9 + 1,7 @@ #"-1,9" means lines 1 to 9 of the first file, "+ 1,7" means lines 1 to 7 of the second file -wang ming qu *5 #The first file must be deleted before it can match the second file, - means delete -wangmingqu*5 -wangmingqu*5 wang ming qu wang ming qu -wang ming + wang ming qu #The first file must be added before it can match the second file, + means adding + wang ming qu wang ming wang ming wang ming @@ -11,3 + 9,7 @@ wang wang wang + wang quan zi zhi + wang quan zi zhi + wang quan zi zhi + wang quan zi zhi
Compare the differences between the two directories
#By default, the contents of the same files in the two directories will also be compared. diff /tmp/ /etc #If you only need to compare the differences between the files in the two directories and do not need to further compare Wenjing content, you need to add the parameter -q diff -q /tmp/ /etc
Other tips
Sometimes we need to use one file as a standard to modify other files, and when there are many modifications, we can do it by patching.
- First find the difference between the files and then output to another file
diff -uN old.txt new.txt > change.txt -u: merge mode display -N: treat non-existing files as empty files
- Patches different content to files
patch old.txt change.txt
- Test verification
diff old.txt new.txt
paste
Introduction
Purpose
paste tool is used to merge lines of files
Grammar
paste option filename1 filename2
Options
-d: Custom separator, the default is tab -s: serial processing,
Help
man paste
Examples
Split and merge two files with:
paste -d ":" old.txt new.txt
Serially merge two files
paste -s old.txt new.txt #Display content: The first line is all the first file; the second line is all the second file
tr
Introduction
Purpose
tr is used for character conversion, replacement and deletion; mainly used to delete control characters in files or perform character conversion.
Grammar
#Usage 1: The execution result of the command is handed over to tr for processing, where string1 is used for query and string2 is used for conversion processing. commands | tr 'string1' 'string2' #Usage 2: The content processed by tr comes from files, remember to use "<" standard input tr 'string1' 'string2' < filename #Usage 3: Match string1 and perform corresponding operations, such as deletion operations tr option 'string1' < filename
Options
-d: Delete all input characters in string 1. -s: Delete all repeated character sequences and keep only the first one; that is, compress the repeated character string into one string.
String
| String | Meaning | Remarks |
|---|---|---|
| a-z or [:lower:] | Match all lowercase letters | |
| A-Z or [:upper:] | Match all uppercase letters | |
| 0-9 or [:digit:] | Match all numbers | |
| [:alnum:] | Match all letters and numbers | |
| [:alpha:] | Match all letters | |
| [:blank:] | Match all horizontal whitespace | |
| [:punct:] | Match all punctuation characters | |
| [:space:] | Match all horizontal or vertical spaces | |
| [:cntrl:] | Match all control characters | \f Ctrl-L Walk page change |
| \ Ctrl-J newline |
||
| \r Ctrl-M Enter | ||
| \t Ctrl-I tab key |
Help
man tr
Examples
Replace all lowercase letters in the file with uppercase letters
tr 'a-z' 'A-Z' < /tmp/passwd tr '[:lower:]' '[:upper:]' < /tmp/passwd
Replace all numbers in the file with special symbols
tr '0-9' '@' < /tmp/passwd tr '[:digit:]' '@' < /tmp/passwd
Replace all “” in the file with “#”
tr ':/' '#' < /tmp/passwd #Note: Character replacement can only be done one by one.
Remove all lowercase letters in the file
tr -d 'a-z' < /tmp/passwd
Delete all characters except letters in the file
tr -d ':/, 0-9[:punct:]' < /tmp/passwd
Small exercise
Idea:
- disassembly task
- Analysis command
- Achieve goals step by step
List the running level of the current system and only display the level number
runlevel | cut -d " " -f 2 runlevel | cut -c 3
Intercept the IP, Netmask, and broadcast address of the current host respectively
ifconfig ens32 | grep inet | grep -v inet6 | awk -F " " '{print $2,$4,$6}' | tr ' ' '\
'
Intercept the MAC address of the current host
ifconfig ens32 | grep 'ether' | tr -s ' ' | cut -d ' ' -f 3