54

I want to search a configuration file for this expression: "central.database". I then want to change the setting associated with "central.database" to "SQLTEST".

The layout of the config file would look like this initially:

central.database = SQLFIRSTTEST

This is what i want it to look like after the sed replacement:

central.database = SQLTEST

I am doing this in a bash script, any suggestions, recommendations or alternative solutions are welcome!

(Actually both central.database and SQLTEST come from bash variables here.)


My current code (third attempt):

sshRetValue=$(ssh -p "35903" -i $HOME/sshids/idrsa-1.old ${1} <<EOF
        sed -i "s/^\($CENTRAL_DB_NAME\s*=\s*\).*\$/\1$CENTRAL_DB_VALUE/" /home/testing.txt;
        echo $?
EOF
)

Error message:

Pseudo-terminal will not be allocated because stdin is not a terminal.
sed: -e expression #1, char 58: unknown option to `s'
-bash: line 3: EOF: command not found
1
  • This is tagged linux but the answers are mostly portable to other platforms. *BSD (including MacOS) users want to use sed -i "" where Linux permits just sed -i. There will also be differences in details of sed syntax between platforms, but simple s/foo/bar/ should be completely portable.
    – tripleee
    Jan 9, 2020 at 5:09

8 Answers 8

120
sed -i -e '/central\.database =/ s/= .*/= new_value/' /path/to/file

Explanation:

  • -i tells sed to save the results to the input file. Without it sed will print the results to stdout.
  • /central\.database =/ matches lines that contain the string between slashes: central.database =. The . is escaped since it's a special character in regex.
  • The s/OLD/NEW/ part performs a substitution. The OLD string is a regular expression to match and the NEW part is the string to substitute in.
  • In regular expressions, .* means "match anything". So = .* matches an equal sign, space, and then anything else afterward.
2
  • The OP's requirement was that "... both central.database and SQLTEST come from bash variables". For example, say CENTRAL_DB_KEY=central.database and CENTRAL_DB_VALUE=SQLTEST. How would you handle adding the escaped dot \. to the first bash variable? (Perhaps ${CENTRAL_DB_KEY//./\\.} would work, but I haven't tested this. What about AIRPORT=$'O\'Hare'?) Dec 3, 2019 at 17:46
  • For anyone that wants to change a key's value under a specific section, for example: [sectionA] central.database = new_val, you can use sed's range addresses to do so: sed -i -e '/\[sectionA\]/, /central\.database =/ s/= .*/= new_value/' /path/to/file
    – Hua2308
    Feb 3, 2022 at 1:37
74

Here's an example expression:

sed -i 's/^\(central\.database\s*=\s*\).*$/\1SQLTEST/' file.cfg

If you want to match stuff with / in it, you can use another delimiter:

sed -i 's#^\(cent/ral\.data/base\s*=\s*\).*$#\1SQL/TEST#' file.cfg

Or with variable expansion:

VAL="SQLTEST"
sed -i "s/^\(central\.database\s*=\s*\).*\$/\1$VAL/" file.cfg

In your example:

sshRetValue=`sed -i "s/^\(\1$CENTRAL_DB_NAME\s*=\s*\).*\$/\1$CENTRAL_DB_VALUE/" /home/testing.txt`;

There's a \1 before $CENTRAL_DB_NAME that's invalid. Also, sed doesn't print it's return value. This is the preferred way to check return values:

sed -i "s/^\($CENTRAL_DB_NAME\s*=\s*\).*\$/\1$CENTRAL_DB_VALUE/" /home/testing.txt;
sed_return_value=$?

And ultimately piping to ssh (not tested):

sed_return_value=$(ssh server <<EOF
    sed -i "s/^\($CENTRAL_DB_NAME\s*=\s*\).*\$/\1$CENTRAL_DB_VALUE/" /home/testing.txt;
    echo $?
EOF
)

The -i is for replacing data in the input file. Otherwise sed writes to stdout.

Regular expressions are a field of their own. It would be impossible to explain them in depth in a stackoverflow answer, unless there is some specific function that's eluding you.

1
  • The OP's requirement was that "... both central.database and SQLTEST come from bash variables". For example, say KEY=central.database and VALUE=SQLTEST. How would you handle adding the escaped dot \. to the bash variable $KEY? (Perhaps ${KEY//./\\.} would work, but I haven't tested this. What about AIRPORT=$'O\'Hare'?) Dec 3, 2019 at 15:20
10

I know it is too late to add an answer to this question however, I thought to share my knowledge to you all. There is a very general approach which I have followed to solve a similar kind of problem. I have deleted the whole line which is matching the string and added the required values to that key. To your question here is the answer

replaceValue=SQLTEST
sed -i "/central.database =/d" /home/testing.txt
echo "central.database = $replaceValue"  >> /home/testing.txt

sed deletes the matching string line from the file and the immediate next line is inserting the required key and value to the file.

2
  • 3
    This is the best answer if you don't know the key already exists. I think the simplicity is nicer and you will have less escaping issues
    – user17130
    Nov 30, 2018 at 20:34
  • Also, the last line will not work with sudo. tee could be used instead: echo "central.database = $replaceValue" | sudo tee -a /home/testing.txt Mar 28, 2020 at 0:55
5

I like using awk for this, since it is quite easy to understand what it is doing and takes care very well of the separator (=) and also the fact that it must be done to an uncommented line:

awk -v var="my_var" -v new_val="NEW VALUE" \  # set the vars
    'BEGIN{FS=OFS="="}                        # set separator to =
     match($1, "^\\s*" var "\\s*") {          # check if it matches
         $2=" " new_val                       # if so, replace the line
     }1' conf_file                            # print all lines

This uses match() to check if the pattern occurs in any given line. If it does, it performs the replacement with the given value.

For example:

$ cat conf
hello
my_var= SOME VALUE
#my_var = ANOTHER VALUE
bye

Let's change the value in my_var to NEW VALUE:

$ awk -v var="my_var" -v new_val="NEW VALUE" 'BEGIN{FS=OFS="="}match($1, "^\\s*" var "\\s*") {$2=" " new_val}1' conf
hello
my_var= NEW VALUE
#my_var = ANOTHER VALUE
bye

It is also possible to set the values in shell variables and then use them with -v:

$ var="my_var"
$ new_value="NEW VALUE"
$ awk -v var="$var" -v new_val="$new_value" 'BEGIN{FS=OFS="="}match($1, "^\\s*" var "\\s*") {$2=" " new_val}1' conf

And you can of course put all of this within a shell function that you then call normally:

#!/bin/bash

replace () {
   file=$1
   var=$2
   new_value=$3
   awk -v var="$var" -v new_val="$new_value" 'BEGIN{FS=OFS="="}match($1, "^\\s*" var "\\s*") {$2=" " new_val}1' "$file"
}

# Call the replace() function with the necessary parameters
replace "conf" "my_var" "NEW VALUE" 

Upon execution, this returns

hello
my_var= NEW VALUE
#my_var = ANOTHER VALUE
bye

While you can also make the script receive the parameters in a way like: ./script.sh "conf_file" "var_to_replace" "NEW VALUE" to then pass them to the function.

3
  • @ThomasWeller of course, see my updated answer with this approach as well.
    – fedorqui
    Aug 29, 2016 at 13:09
  • Unless I am missing something, this matches the beginning only. i.e. replace my_conf.txt "foo" 1 will the replace the value for anything starting with foo.
    – sfrank
    Jun 7 at 13:43
  • As @sfrank mentioned, it takes also parts of variable names, if you add a $ sign at the end of match($1, "^\\s*" var "\\s*$") then it works for me only for entire variable names. I added a | grep $var at the end, so the output is no the whole file, but just relevant line candidates.
    – HaPi
    Jul 11 at 8:10
2

If you want to replace between 2 property files you can use this:

awk -F= 'NR==FNR{A[$1]=$2;next}$1 in A{$2=A[$1]}1' OFS='\=' /tmp/masterfile /opt/props/finalfile.properties > /tmp/tmp.txt && mv -f /tmp/tmp.txt /opt/props/finalfile.properties
2

I have a file called "config.php" and I wanted to change one of its definitions lines.

For example, the line:

define('SOME_CONSTANT', 'old_value');

had to be replaced with this one:

define('SOME_CONSTANT', 'new_value');

So I did that:

sed -i -e "/.*SOME_CONSTANT*./ s/.*/define('SOME_CONSTANT', 'new_value');/" path/to/config.php

In the first part, I am looking for a line that contains "SOME_CONSTANT" (thus the wildcards)

Then I replace that line with a new definition such as: define('SOME_CONSTANT', 'new_value');

Tested and works fine in Centos 7

1

Worked example

Imagine we want to update a Postgres configuration file to increase the number of connections.

We want to update this line in /var/lib/postgresql/data/postgresql.conf:

max_connections = 100

Using sed:

sed -i 's/max_connections = 100/max_connections = 250/g' /var/lib/postgresql/data/postgresql.conf

This is rather useful when updating Docker configuration files as any updates require automating.

0

I used this script for keeping the priorities..

The arguments $1 will have folder in which multiple config files exist. $2 will have properties which need to be replaced in $1 path and sub paths files #3 will have properties which need to be override on top of $2

It also has hidden logic to check for existence of environment variables for the keys exist in $2 and $3 and give priority to that.

i.e If a key exist in environment that would be highest priority. Next to that would $3 and next to that would $1 file.

#!/bin/bash
#Usage is propertyReplacer <CONFIG_FOLDER_PATH> <CONFIG_FILE_2ND_PRIORITY> <CONFIG_FILE_1ST_PRIORITY>
function propertyReplacer() {

  filePathToAct="$1"
  propertiesFilePath="$2"
  propertiesSecureFilePath="$3"

  declare -A keyValues

  while IFS='=' read -r key value; do
    if [  "$key" == "" ]; then
      continue
    elif [[  "$key" =~ ^#.*$ ]]; then
      continue
    else
      echo $key " --> " $value
      keyValues[$key]=$value
    fi
  done < "$propertiesFilePath"

  if [ ! -f "$propertiesSecureFilePath" ]; then
    continue
  else
    while IFS='=' read -r key value; do
      if [  "$key" == "" ]; then
        continue
      elif [[  "$key" =~ ^#.*$ ]]; then
        continue
      else
        echo $key " --> " $value
        keyValues[$key]=$value
      fi
    done < "$propertiesSecureFilePath"
  fi

  for key in ${!keyValues[@]}; do
    envProp=${key//[@]/}
    if [  "$(eval echo '$'$envProp)" == "" ]; then
      echo "Environment key not exist" $envProp
    else
      value=$(eval echo '$'$envProp)
      echo "From Environment " $envProp " --> "$value 
      keyValues[$key]=$value
    fi
  done 


find "$filePathToAct" | while read -r resultFileName; do
  if [ ! -f "$resultFileName" ]; then
    continue
  else
    echo "Acting on the file $resultFileName"

    for key in ${!keyValues[@]}; do
      value=$(echo "${keyValues[${key}]}" | sed 's/\//\\\//g')
      echo "sed -i 's/$key/$value/g' $resultFileName "
      eval "sed -i 's/$key/$value/g' $resultFileName "
    done 
  fi
done 

} 
0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.