Escape a string for a sed replace pattern

In my bash script I have an external (received from user) string, which I should use in sed pattern.

REPLACE='<funny characters here>'
sed 's/KEYWORD/$REPLACE/g'

How can I escape the $REPLACE string so it would be safely accepted by sed as a literal replacement?

NOTE: The KEYWORD is a dumb substring with no matches etc. It is not supplied by user.

Answers:

Here is an example of an AWK I used a while ago. It is an AWK that prints new AWKS. AWK and SED being similar it may be a good template.

ls | awk '{ print 'awk ' '''''''  ' {print $1,$2,$3} ' '''''''  ' ' $1 '.old_ext > ' $1 '.new_ext'  }' > for_the_birds

It looks excessive, but somehow that combination of quotes works to keep the ' printed as literals. Then if I remember correctly the vaiables are just surrounded with quotes like this: '$1'. Try it, let me know how it works with SED.

Answers:

The only three literal characters which are treated specially in the replace clause are / (to close the clause), (to escape characters, backreference, &c.), and & (to include the match in the replacement). Therefore, all you need to do is escape those three characters:

sed 's/KEYWORD/$(echo $REPLACE | sed -e 's/\/\\/g; s///\//g; s/&/\&/g')/g'

Example:

$ export REPLACE='''|\/><&!'
$ echo fooKEYWORDbar | sed 's/KEYWORD/$(echo $REPLACE | sed -e 's/\/\\/g; s///\//g; s/&/\&/g')/g'
foo''|/><&!bar

Answers:

Warning: This does not consider newlines. For a more in-depth answer, see this SO-question instead. (Thanks, Ed Morton & Niklas Peter)

Note that escaping everything is a bad idea. Sed needs many characters to be escaped to get their special meaning. For example, if you escape a digit in the replacement string, it will turn in to a backreference.

As Ben Blank said, there are only three characters that need to be escaped in the replacement string (escapes themselves, forward slash for end of statement and & for replace all):

ESCAPED_REPLACE=$(printf '%s
' '$REPLACE' | sed -e 's/[/&]/\&/g')
# Now you can use ESCAPED_REPLACE in the original sed statement
sed 's/KEYWORD/$ESCAPED_REPLACE/g'

If you ever need to escape the KEYWORD string, the following is the one you need:

sed -e 's/[]/$*.^[]/\&/g'

And can be used by:

KEYWORD='The Keyword You Need';
ESCAPED_KEYWORD=$(printf '%s
' '$KEYWORD' | sed -e 's/[]/$*.^[]/\&/g');

# Now you can use it inside the original sed statement to replace text
sed 's/$ESCAPED_KEYWORD/$ESCAPED_REPLACE/g'

Remember, if you use a character other than / as delimiter, you need replace the slash in the expressions above wih the character you are using. See PeterJCLaw's comment for explanation.

Edited: Due to some corner cases previously not accounted for, the commands above have changed several times. Check the edit history for details.

Answers:

Based on Pianosaurus's regular expressions, I made a bash function that escapes both keyword and replacement.

function sedeasy {
  sed -i 's/$(echo $1 | sed -e 's/([[/.*]|])/\&/g')/$(echo $2 | sed -e 's/[/&]/\&/g')/g' $3
}

Here's how you use it:

sedeasy 'include /etc/nginx/conf.d/*' 'include /apps/*/conf/nginx.conf' /etc/nginx/nginx.conf

Answers:

don't forget all the pleasure that occur with the shell limitation around ' and '

so (in ksh)

Var='>New version of 'content' here <'
printf '%s' '${Var}' | sed 's/[&/\\*\'']/\&/g' | read -r EscVar

echo 'Here is your 'text' to change' | sed 's/text/${EscVar}/g'

Answers:

It's a bit late to respond... but there IS a much simpler way to do this. Just change the delimiter (i.e., the character that separates fields). So, instead of s/foo/bar/ you write s|bar|foo.

And, here's the easy way to do this:

sed 's|/*!50017 DEFINER=`snafu`@`localhost`*/||g'

The resulting output is devoid of that nasty DEFINER clause.

Answers:

Use awk - it is cleaner:

$ awk -v R='//addr:\file' '{ sub('THIS', R, $0); print $0 }' <<< 'http://file:_THIS_/path/to/a/file\is\a\ nightmare'
http://file:_//addr:ile_/path/to/a/file\is\a\ nightmare

Answers:

The sed command allows you to use other characters instead of / as separator:

sed 's#'http://www.fubar.com'#URL_FUBAR#g'

The double quotes are not a problem.

Answers:

It turns out you're asking the wrong question. I also asked the wrong question. The reason it's wrong is the beginning of the first sentence: 'In my bash script...'.

I had the same question & made the same mistake. If you're using bash, you don't need to use sed to do string replacements (and it's much cleaner to use the replace feature built into bash).

Instead of something like, for example:

function escape-all-funny-characters() { UNKNOWN_CODE_THAT_ANSWERS_THE_QUESTION_YOU_ASKED; }
INPUT='some long string with KEYWORD that need replacing KEYWORD.'
A='$(escape-all-funny-characters 'KEYWORD')'
B='$(escape-all-funny-characters '<funny characters here>')'
OUTPUT='$(sed 's/$A/$B/g' <<<'$INPUT')'

you can use bash features exclusively:

INPUT='some long string with KEYWORD that need replacing KEYWORD.'
A='KEYWORD'
B='<funny characters here>'
OUTPUT='${INPUT//'$A'/'$B'}'

Answers:

If the case happens to be that you are generating a random password to pass to sed replace pattern, then you choose to be careful about which set of characters in the random string. If you choose a password made by encoding a value as base64, then there is is only character that is both possible in base64 and is also a special character in sed replace pattern. That character is '/', and is easily removed from the password you are generating:

# password 32 characters log, minus any copies of the '/' character.
pass=`openssl rand -base64 32 | sed -e 's////g'`;

Answers:

If you are just looking to replace Variable value in sed command then just remove Example:

sed -i 's/dev-/dev-$ENV/g' test to sed -i s/dev-/dev-$ENV/g test

Answers:

I have an improvement over the sedeasy function, which WILL break with special characters like tab.

function sedeasy_improved {
    sed -i 's/$(
        echo '$1' | sed -e 's/([[/.*]|])/\&/g' 
            | sed -e 's:	:\t:g'
    )/$(
        echo '$2' | sed -e 's/[/&]/\&/g' 
            | sed -e 's:	:\t:g'
    )/g' '$3'
}

So, whats different? $1 and $2 wrapped in quotes to avoid shell expansions and preserve tabs or double spaces.

Additional piping | sed -e 's: :\t:g' (I like : as token) which transforms a tab in .

Answers:

An easier way to do this is simply building the string before hand and using it as a parameter for sed

rpstring='s/KEYWORD/$REPLACE/g'
sed -i $rpstring  test.txt

Answers:

These are the escape codes that I've found:

* = x2a
( = x28
) = x29

' = x22
/ = x2f
 = x5c

' = x27
? = x3f
% = x25
^ = x5e

Answers: