Find and replace string in a file using 'sed' or 'awk'

To find and replace a string in a file using Linux, use the sed command with the -i flag for in-place editing. The basic syntax is sed -i 's/old_string/new_string/g' filename.txt. This command searches for old_string and replaces all occurrences with new_string directly within the specified file.

Using sed for Simple Substitutions

The sed (Stream Editor) utility is the most common tool for text substitution in Unix-like systems. It processes input streams and can modify files without creating temporary backups if configured correctly. The core operation relies on the substitute command, which follows the pattern s/pattern/replacement/flags. To perform a global replacement, ensuring every instance of the target string is changed rather than just the first one per line, append the g flag. For example, to replace all instances of "error" with "warning" in a log file named app.log, you would run:

sed -i 's/error/warning/g' app.log

The -i option instructs sed to edit files in place. Without this flag, sed outputs the modified content to standard output, allowing you to pipe it into a new file or view it in the terminal. In-place editing modifies the original file, so it is crucial to understand that any data loss is immediate. If you wish to create a backup before modifying the file, you can provide an extension to the -i flag, such as -i.bak. This creates a copy of the original file with the .bak suffix before applying changes. For more complex scenarios involving regular expressions, sed supports extended regex with the -E flag. This allows for more flexible pattern matching, such as replacing dates in various formats. However, for simple string replacements, basic regex is sufficient and often safer to avoid unintended matches.

Handling Special Characters and Escaping

One of the most frequent pitfalls when using sed involves special characters in the search or replacement strings. The delimiter used in the substitute command (typically /) must not appear in the string being searched or replaced unless it is escaped. If your strings contain slashes, such as file paths like /usr/bin, you should either escape them or choose a different delimiter. Using a different delimiter, such as | or #, can simplify commands dealing with paths. For instance:

sed -i 's|/old/path|/new/path|g' config.yaml

This approach eliminates the need to backslash-escape each slash, making the command cleaner and less error-prone. Additionally, if the replacement string contains characters that sed interprets specially, such as & (which refers to the matched pattern) or \, these must be escaped with a backslash. For example, to replace a string with the literal text "test&value", you must write test\&value in the replacement part of the command. Understanding how sed handles these characters prevents subtle bugs where parts of the original text are unexpectedly retained or duplicated in the output. Always test your sed commands on a copy of the file or without the -i flag first to verify the output matches expectations.

Using awk for Advanced Text Processing

While sed excels at line-by-line substitutions, awk offers a more powerful framework for text processing, particularly when you need to manipulate fields or perform conditional logic during replacement. awk treats input as records separated by delimiters, making it ideal for structured data like CSV or TSV files. To replace a specific field in a comma-separated file, you can use awk with the -F option to set the field separator. For example, to change the second column of a CSV file from "old" to "new":

awk -F',' '{if ($2 == "old") $2 = "new"; print}' OFS=',' file.csv > temp.csv && mv temp.csv file.csv

Note that awk does not support in-place editing natively in all versions, so redirecting output to a temporary file and then moving it back is a common workflow. This method ensures data integrity by writing the entire processed file atomically. You can also combine multiple conditions, such as replacing values only in rows where a specific column matches a criterion. For simple string replacements across entire lines, sed is generally faster and requires less code. However, when the replacement depends on the context of other fields or requires arithmetic operations, awk is the superior choice. Both tools complement each other well in shell scripts, allowing you to leverage sed for quick fixes and awk for complex transformations.

When to Choose sed vs awk

Deciding between these tools often comes down to complexity. Use sed for straightforward find-and-replace operations, especially when working with plain text files or logs. Its simplicity makes it easy to script and automate. Use awk when you need to parse columns, apply logic based on field values, or handle structured data formats. If you need to delete words starting with a specific string instead of replacing them, consider looking at our guide on How to Delete All Words Starting with a Specific String in a File. This complements substitution tasks by showing how to remove unwanted patterns entirely.

Common Pitfalls and Best Practices

A common mistake is forgetting the g flag, resulting in only the first occurrence on each line being replaced. Another issue arises when using variables in the sed command; double quotes must be used around the expression to allow variable expansion, whereas single quotes treat the content literally.

OLD="foo"
NEW="bar"
sed -i "s/$OLD/$NEW/g" file.txt

Always validate your commands on a small sample or non-critical file before running them on production data. Creating backups with -i.bak provides a safety net. Additionally, be aware that sed behavior can vary slightly between GNU sed (common on Linux) and BSD sed (common on macOS), particularly regarding the -i flag syntax. On macOS, -i '' is required for in-place editing without backup. For related tasks, such as renaming files after modification, you might find our step-by-step guide on How to Rename a File in Linux: Step-by-Step Guide useful for managing file lifecycle operations.

Frequently Asked Questions

How do I replace a string only on specific lines?

You can specify line numbers or ranges in the sed command. For example, sed -i '10,20 s/old/new/g' file.txt will only replace the string in lines 10 through 20. This is useful for targeted edits in large configuration files.

Can I use sed to replace strings recursively in multiple files?

Yes, you can combine sed with find to process multiple files. Use find . -type f -name "*.txt" -exec sed -i 's/old/new/g' {} + to replace strings in all text files in the current directory and its subdirectories.

What if my search string contains special regex characters?

sed interprets the search pattern as a regular expression. To match literal dots, asterisks, or plus signs, you must escape them with a backslash. For example, to replace "file.txt", use sed 's/file\.txt/newfile.doc/g'.

Why is my sed command not modifying the file?

Ensure you have included the -i flag for in-place editing. Without it, sed prints to stdout. Also, check for typos in the file path and verify that the user has write permissions for the file.

Is awk faster than sed for large files?

Generally, sed is faster for simple substitutions because it is optimized for stream processing. awk has more overhead due to its full programming language features. For massive files with simple find-and-replace needs, sed is the preferred choice for performance.