Dealing with newline characters in environment variables

byGink▻ Sun, 05 Feb 2023

1. Export a variable containing the newline characters in shell script

export MY_VAR="Line 1
Line 2
Line 3"

Note that the newline characters must be specified explicitly within the string. The string must be enclosed in double quotes " in order to preserve the newline characters.

And when getting that variable out, don't forget to enclosed it again:

echo "$MY_VAR"

Pretty easy with shell. Isn't it?

2. Preseve newline characters in `.env` file

Because with .env file we can declare one key=value in one line only. There's no other way but escape newline character by \n. So, if we have some variables with newline characters in this file, .env content should be like this:

MY_VAR="Line 1\nLine 2\nLine 3"

Most of dotenv libraries now are smart enough to convert them back to newline characters when we need $MY_VAR in any programming languages.

3. How to backup exported ENV variables into a `.env` file?

This can be a weird case because exported variable always taking higher priority than .env file in any programming languages. You may think we don't need it, do we?

But believe me, we have to deal with it quite frequently when we need to back up those values. So how to export the environment variable with newline characters to .env file and preserve them in the right format.

Here is the trick:

#!/bin/bash
echo "MY_VAR=${MY_VAR//$'\n'/\\n}" > .env

Let's explain a bit:

#!/bin/bash This is known as a shebang line, which specifies the interpreter for the script. In this case, it specifies that the script should be interpreted by the bash shell. Keep this line as the other shells like (sh, ash...) will work in a different manner.
"${MY_VAR//$'\n'/\\n}" This line prints the contents of the environment variable MY_VAR to the standard output. The $'\n' syntax is used to represent a newline character in a shell string. The expression ${MY_VAR//$'\n'/\\n} uses parameter expansion to replace all occurrences of newline characters in $MY_VAR with \n. The double slashes // indicate that all occurrences of the pattern should be replaced, not just the first one.
> .env This redirection operator writes the output of the echo command to a file named .env instead of printing it to the terminal. If the file does not exist, it will be created. If the file exists, its contents will be overwritten.

Well done. So the variable containing newline characters had been backed up to dotenv correctly and we can use it in any other developing environment.

P/S: If you want to use sh instead of bash. Can try this way:

#!/bin/sh
echo "MY_VAR=$MY_VAR" | sed ':a;N;$!ba;s/\n/\\n/g' > .env

This requires sed command available, which generally can be found in alpine linux along with sh.

And if you want to backup all of variables at once, may try this with bash (version required: ≥5).

#!/bin/bash
declare -p | awk '/^declare -x/{print $3}' | awk -F= '{print $1}' | xargs -I{} bash -c "echo \"{}=\${{}//$'\n'/\\\n}\"" > .env

There's no equivalent for sh anyway, printenv or env can help a bit but then you have to manually adjust newline characters to the right format. 💪