Multiline editing with sed
Published:
sed(1) is notorious for not allowing patterns containing the newline character, but there are ways to work around this limitation.
This post is inspired by that one time when I needed to edit more than a hundred files to insert a line between two other lines, which looks like the perfect job for a pipeline of find(1), xargs(1), and sed(1). The problem is, sed(1) is line-oriented, and does not allow multiline patterns.
Naturally, I went on the internet to see if there is a solution to this conundrum, and I found one stackoverflow. Actually, there were several suggested answers.
Arcane sorcery
The first answer was a small sed program utilising loops, branches, loading lines into the pattern space, and all that. The simplified version looks like this:
/foo$/ { N; s/foo\nbaz/foo\nbar\nbaz/ }
Looks fine, but there are probably edge-cases that can bite you in the ass.
It works by matching the first line of you desired
pattern and, if that worked, executing a block of commands.
Inside the block you load the second line into the pattern
space using N (which loads the immediately following
line into the pattern space), and then executing the substitution
you want.
Simple enough, but I am not even going to entertain the notion of engaging in scripting sed(1) unless absolutely necessary. If this was the only way, I would have just written a short Python script to do the job, but, fortunately, that was not the case.
An abomination
The second answer was simply "Use perl
instead of sed":
perl -pie 's/foo\nbaz/foo\nbar\nbaz/' file.txt
As the caption says, the police will not come haul you to jail if you use the above line, but fuck Perl. It is an abomination, a terribly-designed programming language, created by someone who was more concerned about being cute than correct. I once had the displeasure of working on a system written in Perl, and I learned to hate this language with every ounce of my being, body and soul, because of how atrocious it is. It seems to me as if Perl was designed to deceive and trick programmers using it, setting pitfalls everywhere, and putting loaded footguns wherever a pitfall would not fit. There is no cooperation between you and Perl, there is only struggle and strife. You have to become paranoid and put the language in the heaviest of shackles to achieve anything substantial, but why would you do it to yourself? Just use a different language.
Anyway.
A nice trick
The third answer was to use a tr(1) trick: translate newlines to some other character, run you sed(1) command, then translate back to newlines.
cat file.txt |
tr '\n' '\r' |
sed -e 's/foo\nbaz/foo\nbar\nbaz/' |
tr '\r' '\n'
While nice in theory, it is less useful in practice, because it does not allow modifying files in place. You would have to cook up a more complex pipeline with file renames if you need to modify more than a couple files.
The double tr(1) trick is definitely something to keep up your sleeve if you do not have access to GNU version of sed(1).
A reasonable thing
The fourth answer is the one I liked the most, and
the one that solved my problem.
If you have access to GNU sed(1) (and, in 2026, why would
you not) you can just use the -z switch to change the
line separator to NUL.
With this one simple trick you can just use a simple, easily
xargs(1)-able command:
sed -z -i 's/foo\nbaz/foo\nbar\nbaz/' file.txt
As in most cases, the GNU version of a classic Unix tool comes with some extra quality-of-life features.
Separation by nul
The possibility of switching from newline-separated to nul-separated
records is not unique to sed(1).
Equivalent flags exist for other programs too eg, find(1)
has -print0 to switch from newlines to nuls on output,
and xargs(1) has -0 to accept nul-terminated
input.
This is the most famous pair, but I suspect there exist other
programs with similar mechanisms.
It just makes sense.
nul vs null
Isn't nul a typo? Kind of, yes, but not really.
If you look at ascii(7) you will learn that:
Oct Dec Hex Char
───────────────────────────────────────────
000 0 00 NUL '\0' (null character)
So the name of the character encoded by a byte composed only of zero
bits ie, a null character, is NUL.