Three musketeers of text: all read and process by line.
- grep filters line content.
- awk filter fields.
- sed Filters line contents; modifies line contents.
sed editor
sed is a stream editor that edits data streams based on a pre-provided set of rules before the editor processes the data.
The sed editor can process data in a data stream according to commands entered either from the command line or stored in a command text file.
Sed is mainly used to automatically edit one or more files; simplify repeated operations on files; write conversion programs, etc.
sed workflow
The workflow of sed mainly includes three processes of reading, executing and displaying:
- Read: sed reads a line from the input stream (file, pipe, standard input) and stores it in a temporary buffer (also known as pattern space, pattern space).
- Execution: By default, all sed commands are executed sequentially in the pattern space. Unless a line address is specified, the sed command will be executed sequentially on all lines.
- Display: Send the modified content to the output stream. After sending data, the pattern space will be cleared. Before all the contents of the file are processed, the above process will be repeated until all the contents are processed.
Before all the contents of the file are processed, the above process will be repeated until all the contents are processed.
Note: By default, all sed commands are executed in the pattern space, so the input file will not change, unless the source file is modified with “sed -i”, or the output is redirected to a new in the file.
sed command format
sed -e 'action' file1 file2 sed -n -e 'operation' file1 file2 sed -f script file file1 file2 sed -i -e 'operation' file 1 file 2
Three ways to execute multiple commands
sed -n -e 'action 1' -e 'action 2' file
?
sed -n -e 'operation 1; operation 2' file
?
sed -e 'n{
operation 1
operation 2
?…
}' file 1
Common options:
| Option | Action |
|---|---|
| -e or –expression= | indicates that the specified command is used to process the input text file, and it can be omitted when there is only one operation command. Generally, when executing multiple operation commands, use |
| -f or –file= | means to use the specified script file to process the input text file |
| -h or –help | show help |
| -n, –quiet or –silent | Suppress sed editor output, but can be used with p command to complete output |
| -i | Directly modify the target text file |
Common actions:
| operation | function |
|---|---|
| s | Replace, replace the specified character |
| d | Delete, delete the selected line |
| a | Increase, add a line of specified content below the current line |
| i | Insert, insert a line of specified content above the selected line |
| c | Replace, replace the selected line with the specified content |
| y | Character conversion, the character length before and after conversion must be the same |
| p | Print line content. If the line is specified at the same time, it means to print the specified line; if no line is specified, it means to print all the content; if there are non-printing characters, it will be output in ASCII code. It is usually used with the “-n” option |
| = | print line number |
| l (Lowercase L) | Print the text and non-printable ASCII characters in the data stream (such as the end character $, tab character \t) |
The core function of sed: add, delete, modify and check (can cooperate with regular expressions)
check: p ? delete: d ? Change: s (string replacement), c (whole line replacement), y (corresponding character replacement, the effect is similar to the tr command) ? Increase: i (insert content before the line), a (add content after the line), r (read the content of the file after the line) ? Copy and paste: H (copy), d (delete), G (paste below the specified line)
sed finds and prints p
4.1 No row specified (each row is processed sequentially)
1. The sed editor outputs the line content by default, and the -n option can disable the output. If you do not add -n, but use the p operation, then each line will be printed twice.
sed -e “p” : print each line twice.
sed -n “p” : Print each line only once.
[root@yuji ~]# cat ff.txt //View the content of the file, a total of 10 lines one two three four five six seven eight nine ten [root@yuji ~]# sed -e 'p' ff.txt //The contents of each line will be printed twice one one two two three three four four five five six six seven seven eight eight nine nine ten ten [root@yuji ~]# sed -n -e 'p' ff.txt //The content of each line is only printed once one two three four five six seven eight nine ten
2, ‘n’ prints the line number.
sed -n ‘=’ : print only line numbers.
sed -e ‘n’ : Print line number and line content.
sed -n ‘=;p’ : Print line number and line content.
[root@yuji ~]# sed -e '=' ff.txt //print line number and line content 1 one 2 two 3 three 4 four 5 five 6 six 7 seven 8 eight 9 nine 10 ten
Three ways sed can execute multiple commands.
Take printing line number and line content as an example:
#method one sed -n -e '=' -e 'p' file.txt ? #Method Two sed -n -e '=;p' file.txt ? #Method 3: Newline operation sed -n ' = p 'file.txt
‘l’ prints text ie hidden characters (terminal $, tab \t).
[root@yuji ~]# sed -n 'l' ff.txt one $ two $ three $ four $ five $ six $ seven $ eight $ nine$ ten$
sed operates on the specified line
Two ways:
- Represent the row interval in numerical form;
- Use text patterns (strings) to filter rows (usually combined with regular expressions).
Numerically represent line intervals:
| Operation | Meaning |
|---|---|
| ‘1p’ | Print the first line |
| ‘$p’ | Print the last line |
| ‘1 ,3p’ | Print consecutive lines, print the first line to the third line |
| ‘6,$p’ | print The sixth line to the last line |
| ‘1, + 3p’ | Print the first line plus the next three lines (that is, print the first to fourth lines ) |
| ‘5q’ | Exit after printing the first five lines |
| ‘p;n’ | Print odd lines |
| ‘n;p’ | Print even lines |
Use a string to match lines:
| Operation | Meaning |
|---|---|
| ‘/root/p’ | Print lines that contain root |
| ‘/root/!p’ | Print lines that do not contain root. ! Indicates negation |
| ‘/^root/p’ | Print the line starting with root |
| ‘/bash$/’ | Print the line ending with bash |
| ‘/root l bash/p’ | Print lines containing root or bash. “l” is the metacharacter of the extended regular expression, use sed -r |
| ‘6,/root/p’ | to print line 6 to the first line containing root |
Use regular expressions to match line content
First copy the /etc/passwd file to the home directory and rename it to pass.txt for easy demonstration
[root@yuji ~]# cp /etc/passwd pass.txt //Copy the file and rename it [root@yuji ~]# sed -n '/root/p' pass.txt //Print the line containing root root:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/root:/sbin/nologin [root@yuji ~]# sed -n '/^root/p' pass.txt //Print lines starting with root root:x:0:0:root:/root:/bin/bash [root@yuji ~]# sed -n '/bash$/p' pass.txt //Print the line ending with bash root:x:0:0:root:/root:/bin/bash yuji2:x:1000:1000:yuji2:/home/yuji2:/bin/bash nancy:x:1021:1021::/home/nancy:/bin/bash helen:x:1022:1022::/home/helen:/bin/bash
[root@yuji ~]# sed -n '6,/root/p' pass.txt // start printing from line 6 until the first line containing root sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail:/sbin/nologin operator:x:11:0:operator:/root:/sbin/nologin [root@yuji ~]# sed -n '6,/root/=' pass.txt // Print the line number from line 6 to the first line containing root 6 7 8 9 10
sed -r supports extended regular expressions. At the same time, when using {n}, {n,}, {n,m}, there is no need to add a backslash \ before the bracket {}.
[root@yuji ~]# sed -n '/ro{2,}t/p' pass.txt
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
[root@yuji ~]# sed -n -r '/ro{2,}t/p' pass.txt
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
?
# print lines containing root or bash
[root@yuji ~]# sed -n '/root|bash/p' pass.txt //If -r is not added, then "|" must be preceded by an escape character \
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
yuji2:x:1000:1000:yuji2:/home/yuji2:/bin/bash
nancy:x:1021:1021::/home/nancy:/bin/bash
helen:x:1022:1022::/home/helen:/bin/bash
[root@yuji ~]# sed -n -r '/root|bash/p' pass.txt //"sed -r" supports extended regular expressions
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
yuji2:x:1000:1000:yuji2:/home/yuji2:/bin/bash
nancy:x:1021:1021::/home/nancy:/bin/bash
helen:x:1022:1022::/home/helen:/bin/bash
Match the desired line through the string
[root@yuji ~]# sed '/e$/d' ff.txt //Delete lines ending with e two four six seven eight ten [root@yuji ~]# sed '/e$/!d' ff.txt //Delete all lines except those ending with e one three five
‘/String1/,/String2/d’: Turn on the delete function from the first matching position, and turn off the delete function after deleting from the second matching position. Then continue to search and delete according to this rule.
[root@yuji ~]# cat 2.txt 111 222 333 444 555 111 222 333 444 555 [root@yuji ~]# sed '/2/,/4/d' 2.txt 111 555 111 555 [root@yuji ~]# sed '/1/,/3/d' 2.txt 444 555 444 555
Delete blank lines and save: sed -i ‘/^$/d’ file.txt (backup the target file before using -i)
[root@yuji ~]# cat file.txt 11 ? twenty two ? 33 [root@yuji ~]# sed -i '/^$/d' file.txt [root@yuji ~]# cat file.txt 11 twenty two 33
Three ways to delete blank lines:
- grep -v “^$” file.txt //filter out non-empty lines
- cat file.txt |tr -s “\
” //compress line breaks - sed ‘/^$/d’ file.txt //delete blank lines
sed modify and replace s c y
s: replacement string
c: replace the whole line
y: character replacement, the length of the string before and after replacement must be the same
line range s/old string/new string/replacement token ? #4 replacement tags: Number: Indicates which matching place the new string will replace g: surface new string will replace all matches p: print lines matching the substitution command, used with -n w file: write the replacement result to the file
[root@yuji ~]# sed -n '/root/p' pass.txt //Print all lines containing root root:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/root:/sbin/nologin [root@yuji ~]# sed -n 's/root/aaa/p' pass.txt //Replace the first root of the matching line with aaa aaa:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/aaa:/sbin/nologin [root@yuji ~]# sed -n 's/root/aaa/2p' pass.txt //Replace the second root of the matching line with aaa root:x:0:0:aaa:/root:/bin/bash [root@yuji ~]# sed -n 's/root/aaa/3p' pass.txt //Replace the third root of the matching line with aaa root:x:0:0:root:/aaa:/bin/bash [root@yuji ~]# sed -n 's/root/aaa/gp' pass.txt //Replace all roots in matching lines with aaa aaa:x:0:0:aaa:/aaa:/bin/bash operator:x:11:0:operator:/aaa:/sbin/nologin
[root@yuji ~]# sed -n 's/root//gp' pass.txt //Delete all matching lines as root :x:0:0::/:/bin/bash operator:x:11:0:operator:/:/sbin/nologin [root@yuji ~]# echo 000010101 | sed 's/^0*//' //Delete all 0 at the beginning 10101
[root@yuji ~]# sed -n ‘s/root//gp’ pass.txt //Delete all matching lines as root :x:0:0::/:/bin/bash operator:x:11:0: operator:/:/sbin/nologin [root@yuji ~]# echo 000010101 | sed ‘s/^0*//’
//Delete all 0 10101 at the beginning
[root@yuji ~]# sed -n '/^root/p' pass.txt //Print all lines starting with root root:x:0:0:root:/root:/bin/bash [root@yuji ~]# sed -n '/^root/ s/^/#/p' pass.txt //Filter out the lines starting with root, add # at the beginning of the line #root:x:0:0:root:/root:/bin/bash ? [root@yuji ~]# sed -n '/root/p' pass.txt //Print the line containing root root:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/root:/sbin/nologin [root@yuji ~]# sed -n '/root/ s/$/#/p' pass.txt //Add # at the end of the line containing root root:x:0:0:root:/root:/bin/bash# operator:x:11:0:operator:/root:/sbin/nologin#
Another alternative
Comment out the line containing root content:
Use regular expressions to match the entire line containing root, and then add # before the entire line. & amp; represents the entire line matched by the previous regular expression.
[root@yuji ~]# sed -n '/root/ s/^/#/p' pass.txt //Filter out the lines containing root, add # at the beginning of the line #root:x:0:0:root:/root:/bin/bash #operator:x:11:0:operator:/root:/sbin/nologin [root@yuji ~]# sed -n 's/.*root.*/# & amp;/p' pass.txt #root:x:0:0:root:/root:/bin/bash #operator:x:11:0:operator:/root:/sbin/nologin
[root@yuji ~]# vim 1.sh [root@yuji ~]# cat 1.sh /root/s/^/#/p 3,5 s/$/#/p ? [root@yuji ~]# sed -n -f 1.sh pass.txt //Specify 1.sh file to execute #root:x:0:0:root:/root:/bin/bash daemon:x:2:2:daemon:/sbin:/sbin/nologin# adm:x:3:4:adm:/var/adm:/sbin/nologin# lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin# #operator:x:11:0:operator:/root:/sbin/nologin
: specify the delimiter
When the string contains “/”, you need to add the escape character \ in front to avoid confusion with the separator “/”. This operation is very cumbersome, and it is easy to make mistakes. At this time, you can specify the separator yourself.
The first character after s is the delimiter, and the three delimiters must be consistent. If you encounter a character that is the same as the delimiter, you need to use \ to escape it into a normal character.
#The default delimiter is "/" [root@yuji ~]# sed -n 's//bin/bash//sbin/nologin/p' pass.txt root:x:0:0:root:/root:/sbin/nologin yuji2:x:1000:1000:yuji2:/home/yuji2:/sbin/nologin nancy:x:1021:1021::/home/nancy:/sbin/nologin helen:x:1022:1022::/home/helen:/sbin/nologin ? #Specify # as a separator [root@yuji ~]# sed -n 's#/bin/bash#/sbin/nologin#p' pass.txt root:x:0:0:root:/root:/sbin/nologin yuji2:x:1000:1000:yuji2:/home/yuji2:/sbin/nologin nancy:x:1021:1021::/home/nancy:/sbin/nologin helen:x:1022:1022::/home/helen:/sbin/nologin ? #Specify ! as a delimiter [root@yuji ~]# sed -n 's!/bin/bash!/sbin/nologin!p' pass.txt root:x:0:0:root:/root:/sbin/nologin yuji2:x:1000:1000:yuji2:/home/yuji2:/sbin/nologin nancy:x:1021:1021::/home/nancy:/sbin/nologin helen:x:1022:1022::/home/helen:/sbin/nologin ? #Specify 8 as the delimiter [root@yuji ~]# sed -n 's8/bin/bash8/sbin/nologin8p' pass.txt root:x:0:0:root:/root:/sbin/nologin yuji2:x:1000:1000:yuji2:/home/yuji2:/sbin/nologin nancy:x:1021:1021::/home/nancy:/sbin/nologin helen:x:1022:1022::/home/helen:/sbin/nologin
[root@yuji ~]# cat ff.txt one two three four five six seven eight nine ten [root@yuji ~]# sed '/fo/c 22' ff.txt //Replace the line containing fo with 22 as a whole one two three twenty two five six seven eight nine
Single character replacement y
Using y is to replace a single character, each character needs one-to-one correspondence, not a whole replacement. The length of the string before and after must be the same, otherwise an error will be reported.
Example:
When encountering n, replace it with 2, and when encountering o, replace it with 5.
[root@yuji ~]# sed ‘y/no/25/’ ff.txt //When encountering n, replace it with 2, and when encountering o, replace it with 5 52e tw5 three f5ur five six seve2 eight 2i2e te2 [root@yuji ~] # sed ‘y/no/255/’ ff.txt sed: -e expression #1, character 9: String lengths differ for “y” commands
sed adds a i r
a: add content after the line
i: Insert content before the line
r: read the file content after the line
[root@yuji ~]# sed '1,3a 22' ff.txt //In the 1st~3rd line, add a line of content below each line one twenty two two twenty two three twenty two four five six seven eight nine ten
[root@yuji ~]# cat hh.txt hello world [root@yuji ~]# sed '6r hh.txt' ff.txt //Read the content of the hh.txt file below line 6 one two three four five six hello world seven eight nine ten
copy and paste
Compare vi/vim editor and sed editor
#vi//vim editor: command mode dd p cut paste yy p copy paste last line mode :1,3 co 10 copy and paste (copy and paste lines 1~3 below line 10) :1,3 m 10 Cut and paste (cut lines 1~3 below line 10) ? #sed command: H to copy, d to delete, G to paste below the specified line
[root@yuji ~]# sed '1,3 {H;d};$G' ff.txt //Cut and paste lines 1~3 below the last line
four
five
six
seven
eight
nine
ten
?
one
two
three
[root@yuji ~]# sed '1,3 {H;d};5G' ff.txt //Cut and paste lines 1~3 to the end of line 5
four
five
?
one
two
three
six
seven
eight
nine
ten
The strings in the file content are exchanged
#Swap the two strings [root@yuji ~]# echo 111222333 111222333 [root@yuji ~]# echo 111222333|sed -r 's/(111)(222)/\2\1/' 222111333 ? # Swap the positions of the 3 strings [root@yuji ~]# echo 111222333|sed -r 's/(111)(222)(333)/\3\2\1/' 333222111 ? # swap the first character with the last character [root@yuji ~]# echo 111222333|sed -r 's/^(.)(.*)(.)$/\3\2\1/' 311222331