sed

The following example shows a typical, and the most common, use of sed: substitution. This usage was indeed the original motivation for sed:[4]

sed 's/regexp/replacement/g' inputFileName > outputFileName

In some versions of sed, the expression must be preceded by -e to indicate that an expression follows. The s stands for substitute, while the g stands for global, which means that all matching occurrences in the line would be replaced. The regular expression (i.e. pattern) to be searched is placed after the first delimiting symbol (slash here) and the replacement follows the second symbol. Slash (/) is the conventional symbol, originating in the character for "search" in ed, but any other could be used to make syntax more readable if it does not occur in the pattern or replacement; this is useful to avoid "leaning toothpick syndrome".

The substitution command, which originates in search-and-replace in ed, implements simple parsing and templating. The regexp provides both pattern matching and saving text via sub-expressions, while the replacement can be either literal text, or a format string containing the characters & for "entire match" or the special escape sequences \1 through \9 for the nth saved sub-expression. For example, sed -r "s/(cat|dog)s?/\1s/g" replaces all occurrences of "cat" or "dog" with "cats" or "dogs", without duplicating an existing "s": (cat|dog) is the 1st (and only) saved sub-expression in the regexp, and \1 in the format string substitutes this into the output.

Besides substitution, other forms of simple processing are possible, using some 25 sed commands. For example, the following uses the d command to filter out lines that only contain spaces, or only contain the end of line character:

sed '/^ *$/d' inputFileName

This example uses some of the following regular expression metacharacters (sed supports the full range of regular expressions):

• The caret (^) matches the beginning of the line.
• The dollar sign ($) matches the end of the line.
• The asterisk (*) matches zero or more occurrences of the previous character.
• The plus (+) matches one or more occurrence(s) of the previous character.
• The question mark (?) matches zero or one occurrence of the previous character.
• The dot (.) matches exactly one character.

Complex sed constructs are possible, allowing it to serve as a simple, but highly specialized, programming language. Flow of control, for example, can be managed by the use of a label (a colon followed by a string) and the branch instruction b, as well as the conditional branch t. An instruction b followed by a valid label name will move processing to the command following that label. The t instruction will only do so if there was a successful substitution since the previous t (or the start of the program, in case of the first t encountered). Additionally, the { instruction starts a subsequence of commands (up to the matching }); in most cases, it will be conditioned by an address pattern.

Under Unix, sed is often used as a filter in a pipeline:

$ generateData | sed 's/x/y/g'

That is, a program such as "generateData" generates data, and then sed makes the small change of replacing x with y. For example:

$ echo xyz xyz | sed 's/x/y/g'
yyz yyz

[notes 1]

It is often useful to put several sed commands, one command per line, into a script file such as subst.sed, and then use the -f option to run the commands (such as s/x/y/g) from the file:

sed -f subst.sed inputFileName > outputFileName

Any number of commands may be placed into the script file, and using a script file also avoids problems with shell escaping or substitutions.

Such a script file may be made directly executable from the command line by prepending it with a "shebang line" containing the sed command and assigning the executable permission to the file. For example, a file subst.sed can be created with contents:

#!/bin/sed -f
s/x/y/g

The file may then be made executable by the current user with the chmod command:

chmod u+x subst.sed

The file may then be executed directly from the command line:

subst.sed inputFileName > outputFileName

The -i option, introduced in GNU sed, allows in-place editing of files (actually, a temporary output file is created in the background, and then the original file is replaced by the temporary file). For example:

sed -i 's/abc/def/' fileName

# convert input text stream to "Hello, world!"
s/.*/Hello, world!/
q

This "Hello, world!" script is in a file (e.g., script.txt) and invoked with sed -f script.txt inputFileName, where "inputFileName" is the input text file. The script changes "inputFileName" line #1 to "Hello, world!" and then quits, printing the result before sed exits. Any input lines past line #1 are not read, and not printed. So the sole output is "Hello, world!".

The example emphasizes many key characteristics of sed:

• Typical sed programs are rather short and simple.
• sed scripts can have comments (the line starting with the # symbol).
• The s (substitute) command is the most important sed command.
• sed allows simple programming, with commands such as q (quit).
• sed uses regular expressions, such as .* (zero or more of any character).

Below follow various sed scripts; these can be executed by passing as an argument to sed, or put in a separate file and executed via -f or by making the script itself executable.

To replace any instance of a certain word in a file with "REDACTED", such as an IRC password, and save the result:

$ sed -i "s/yourpassword/REDACTED/" ./status.chat.log

To delete any line containing the word "yourword" (the address is '/yourword/'):

/yourword/ d

To delete all instances of the word "yourword":

s/yourword//g

To delete two words from a file simultaneously:

s/firstword//g
s/secondword//g

To express the previous example on one line, such as when entering at the command line, one may join two commands via the semicolon:

$ sed "s/firstword//g; s/secondword//g" inputFileName

In the next example, sed, which usually only works on one line, removes newlines from sentences where the second line starts with one space. Consider the following text:

This is my dog,
 whose name is Frank.
This is my fish,
whose name is George.
This is my goat,
 whose name is Adam.

The sed script below will turn the text above into the following text. Note that the script affects only the input lines that start with a space:

This is my dog, whose name is Frank.
This is my fish,
whose name is George.
This is my goat, whose name is Adam.

The script is:

 N
 s/\n / /
 P
 D

This is explained as:

• (N) add the next line to the pattern space;
• (s/\n / /) find a new line followed by a space, replace with one space;
• (P) print the top line of the pattern space;
• (D) delete the top line from the pattern space and run the script again.

This can be expressed on a single line via semicolons:

sed 'N; s/\n / /; P; D' inputFileName