Search and Replace Lines with Sed: Master Advanced Techniques

Need to find and replace text in files? Sed’s s command handles this in one line. Good for config updates, data cleaning, or quick edits without opening an editor.

What is sed?

sed (Stream Editor) processes text line by line without loading the whole file into memory. Good for:

• Automation and scripts
• Large files
• Piping with other Unix tools
• Batch processing

Other sed guides:

• Delete lines
• Insert or append text
• Transform text case

Basic Syntax

sed 's/old/new/' file.txt        # Replace first occurrence per line
sed 's/old/new/g' file.txt       # Replace all occurrences (global)
sed 's/old/new/i' file.txt       # Ignore case

Delimiters

Use different delimiters when your pattern contains slashes:

sed 's|/old/path|/new/path|g' config.txt
sed 's#http://old#https://new#g' urls.txt

Key Options

Option	Function	Example
-i	Edit files in-place	sed -i 's/old/new/g' file.txt
-i.bak	Edit in-place with backup	sed -i.bak 's/old/new/g' file.txt
-n	Suppress default output	sed -n 's/old/new/p' file.txt
-e	Multiple commands	sed -e 's/old/new/' -e 's/foo/bar/' file.txt

Important: Test without -i first. Sed outputs to stdout by default.

Common Examples

Config files:

sed 's/localhost/production-server/g' config.ini
sed 's/port=8080/port=80/g' server.conf

Code refactoring:

sed 's/oldFunction/newFunction/g' script.js
sed 's/var /let /g' legacy.js

Data cleanup:

sed 's/,/;/g' data.csv              # Change delimiter
sed 's/  / /g' document.txt         # Fix double spaces
sed 's/\t/ /g' file.txt             # Tabs to spaces

File Operations

sed 's/old/new/g' file.txt > newfile.txt   # Save to new file
sed -i 's/old/new/g' file.txt              # Edit in place
sed -i.bak 's/old/new/g' file.txt          # Edit with backup

Multiple Replacements

sed -e 's/old1/new1/g' -e 's/old2/new2/g' file.txt
sed 's/old1/new1/g; s/old2/new2/g' file.txt

Safety Tips

1. Test without -i first
2. Use -i.bak for backups
3. Verify with diff original.txt modified.txt

Advanced Pattern Matching with Regular Expressions

Regular expressions unlock sed’s full potential for complex search and replace operations. Master these patterns to handle sophisticated text transformations.

Basic Regular Expression Elements

Wildcard character (.):

sed 's/t.st/test/g' filename        # Matches "test", "tast", "t3st", etc.
sed 's/c.t/cat/g' pets.txt          # Matches "cat", "cut", "cot", etc.

Character classes:

sed 's/[aeiou]/X/g' filename        # Replace any vowel with X
sed 's/[0-9]/N/g' filename          # Replace any digit with N
sed 's/[A-Z]/L/g' filename          # Replace uppercase letters with L

Predefined character classes:

sed 's/[[:digit:]]/N/g' filename    # Replace digits (same as [0-9])
sed 's/[[:alpha:]]/L/g' filename    # Replace letters
sed 's/[[:space:]]/X/g' filename    # Replace whitespace characters

Quantifiers

Zero or more (*):

sed 's/ab*c/X/g' filename          # Matches "ac", "abc", "abbc", "abbbc"
sed 's/[0-9]*/NUM/g' filename      # Matches empty string or any digits

One or more (\+):

sed 's/[0-9]\+/NUM/g' filename     # Matches one or more digits
sed 's/a\+/A/g' filename           # Matches "a", "aa", "aaa", etc.

Exact occurrences (\{n\}):

sed 's/[0-9]\{3\}/XXX/g' filename  # Matches exactly 3 digits
sed 's/a\{2,4\}/A/g' filename      # Matches 2 to 4 'a' characters

Anchors and Boundaries

Line anchors:

sed 's/^Error/WARNING/' filename    # Replace "Error" at line start
sed 's/end$/END/' filename          # Replace "end" at line end
sed 's/^$/EMPTY/' filename          # Replace empty lines

Word boundaries:

sed 's/\bcat\b/dog/g' filename      # Replace whole word "cat" only
sed 's/\btest\b/exam/g' filename    # Avoids matching "testing" or "retest"

Practical Advanced Examples

Phone number formatting:

# Transform (123) 456-7890 to 123-456-7890
sed 's/(\([0-9]\{3\}\)) \([0-9]\{3\}\)-\([0-9]\{4\}\)/\1-\2-\3/g' contacts.txt

Email extraction and masking:

# Replace email addresses with [EMAIL]
sed 's/[a-zA-Z0-9._%+-]\+@[a-zA-Z0-9.-]\+\.[a-zA-Z]\{2,\}/[EMAIL]/g' data.txt

Date format conversion:

# Convert MM/DD/YYYY to YYYY-MM-DD
sed 's/\([0-9]\{2\}\)\/\([0-9]\{2\}\)\/\([0-9]\{4\}\)/\3-\1-\2/g' dates.txt

URL protocol updates:

# Change HTTP to HTTPS
sed 's/http:\/\/\([^[:space:]]*\)/https:\/\/\1/g' urls.txt

Grouping and Back-references

Capture groups with \(\) and back-references with \1, \2:

# Swap first and last names
sed 's/\([A-Za-z]*\) \([A-Za-z]*\)/\2, \1/g' names.txt

# Duplicate words detection and removal
sed 's/\b\([a-zA-Z]\+\) \1\b/\1/g' text.txt

# Extract filename from path
sed 's/.*\/\([^\/]*\)$/\1/' paths.txt

Complex Pattern Examples

Log processing:

# Extract timestamp from log entries
sed 's/^\[\([0-9-: ]*\)\] .*/\1/' server.log

# Replace IP addresses with [IP]
sed 's/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/[IP]/g' access.log

Code refactoring:

# Update function calls: oldFunc(param) -> newFunc(param)
sed 's/oldFunc(\([^)]*\))/newFunc(\1)/g' code.js

# Convert single quotes to double quotes in strings
sed "s/'\([^']*\)\"/\"\1\"/g" script.js

Extended Regular Expressions

Using -E flag for enhanced patterns:

# Multiple alternatives with |
sed -E 's/(cat|dog|bird)/animal/g' pets.txt

# Simplified quantifiers (no escaping needed)
sed -E 's/[0-9]{3}-[0-9]{2}-[0-9]{4}/XXX-XX-XXXX/g' ssn.txt

# Non-capturing groups
sed -E 's/(http|https)://[^[:space:]]*/[URL]/g' text.txt

Testing and Debugging Regular Expressions

Preview matches before replacement:

# Show what would be matched
grep 'pattern' filename

# Show line numbers with matches
grep -n 'pattern' filename

# Test with sed's print flag
sed -n 's/pattern/replacement/p' filename

Build patterns incrementally:

# Start simple
sed 's/[0-9]/X/' filename

# Add complexity gradually
sed 's/[0-9]\+/NUM/' filename

# Final complex pattern
sed 's/[0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\}/XXX-XX-XXXX/' filename

Pro tip: Regular expressions can be tricky. Always test your patterns thoroughly on sample data before applying to important files.

Advanced sed Techniques and Best Practices

Master these advanced techniques to make your sed operations more efficient, precise, and safe.

Targeted Line Processing

Limit operations to specific line ranges:

sed '1,10s/old/new/g' file.txt      # Replace only in lines 1-10
sed '5,$s/old/new/g' file.txt       # Replace from line 5 to end
sed '10s/old/new/g' file.txt        # Replace only on line 10

Target lines by pattern:

sed '/pattern/s/old/new/g' file.txt     # Replace only in lines containing "pattern"
sed '/^#/s/old/new/g' file.txt          # Replace only in comment lines
sed '/ERROR/s/old/new/g' log.txt        # Replace only in error lines

Preview and Testing Techniques

Preview changes before applying:

sed -n 's/old/new/p' file.txt           # Show only changed lines
sed 's/old/new/g' file.txt | head -20   # Preview first 20 lines
sed 's/old/new/g' file.txt | diff file.txt -  # Show differences

Test with line numbers:

nl file.txt | sed 's/old/new/g'        # Show line numbers for context

Working with Special Characters

Escape literal characters:

sed 's/\./DOT/g' file.txt              # Escape literal dots
sed 's/\*/STAR/g' file.txt             # Escape literal asterisks
sed 's/\$/DOLLAR/g' file.txt           # Escape literal dollar signs
sed 's/\//SLASH/g' file.txt            # Escape literal forward slashes

Use alternative delimiters:

sed 's|/old/path|/new/path|g' file.txt     # Use | for paths
sed 's#http://old#https://new#g' file.txt  # Use # for URLs
sed 's@old@new@g' file.txt                 # Use @ as delimiter

Batch Processing Multiple Files

Process all files of a type:

find . -name "*.txt" -exec sed -i 's/old/new/g' {} \;
find . -name "*.conf" -exec sed -i.bak 's/old/new/g' {} \;

Using xargs for efficiency:

find . -name "*.txt" | xargs sed -i 's/old/new/g'
find . -name "*.js" -print0 | xargs -0 sed -i 's/console.log/logger.debug/g'

Loop through files:

for file in *.txt; do
    sed -i.backup 's/old/new/g' "$file"
    echo "Processed: $file"
done

Advanced Pattern Techniques

Using back-references for complex replacements:

# Swap two words
sed 's/\(foo\) \(bar\)/\2 \1/g' file.txt

# Duplicate text
sed 's/\(important\)/\1 \1/g' file.txt

# Rearrange data fields
sed 's/\([^,]*\),\([^,]*\),\([^,]*\)/\3,\1,\2/' data.csv

Multiple operations in sequence:

sed -e 's/old1/new1/g' -e 's/old2/new2/g' -e 's/old3/new3/g' file.txt

Conditional replacements:

# Replace only if line contains specific pattern
sed '/contains_this/{s/old/new/g;}' file.txt

# Replace in specific sections
sed '/START/,/END/{s/old/new/g;}' file.txt

Performance Optimization

Process large files efficiently:

# Stop after first match per line (faster)
sed 's/old/new/' file.txt

# Use specific patterns to reduce processing
sed '/pattern/s/old/new/g' file.txt

Combine operations to reduce passes:

# Instead of multiple sed calls
sed 's/old1/new1/g; s/old2/new2/g; s/old3/new3/g' file.txt

Safety and Backup Strategies

Always backup important files:

cp original.txt original.txt.backup
sed -i.$(date +%Y%m%d) 's/old/new/g' original.txt

Test on sample data first:

head -100 largefile.txt > sample.txt
sed 's/old/new/g' sample.txt          # Test your pattern
# If good, apply to original:
sed -i.backup 's/old/new/g' largefile.txt

Use version control:

git add file.txt                      # Stage current version
sed -i 's/old/new/g' file.txt         # Make changes
git diff                              # Review changes

Practical Workflow Examples

Configuration file updates:

# Update server configuration across multiple files
find /etc/nginx -name "*.conf" -exec sed -i.backup \
    -e 's/old-server.com/new-server.com/g' \
    -e 's/port 8080/port 80/g' {} \;

Code refactoring:

# Update function names in JavaScript files
find ./src -name "*.js" -exec sed -i \
    's/\boldFunction\b/newFunction/g' {} \;

Log processing:

# Clean and standardize log files
sed -e 's/DEBUG/[DEBUG]/g' \
    -e 's/ERROR/[ERROR]/g' \
    -e 's/INFO/[INFO]/g' app.log > standardized.log

Common Pitfalls to Avoid

1. Forgetting to escape special characters
2. Not testing patterns before applying to important files
3. Using global replacement when you only want first occurrence
4. Not backing up files before in-place editing
5. Making patterns too broad (matching unintended text)

Pro Tips

• Start with simple patterns and add complexity gradually
• Use grep to test your patterns before using in sed
• Keep a collection of tested sed patterns for reuse
• Document complex regular expressions for future reference
• Consider using awk or perl for very complex text processing

Remember: The key to mastering sed is practice and patience. Build your skills incrementally and always prioritize data safety.

Conclusion

Mastering sed for searching and replacing text is a game-changer for anyone who works with text files regularly. The tricks I’ve shared are just the start to harnessing the full potential of this powerful stream editor. Remember to craft your commands with precision and always double-check your patterns. Whether you’re tweaking a single file or tackling multiple files at once, sed can be your best ally—just be sure to back up your data before you dive in. With these strategies in your toolkit, you’re well on your way to becoming a sed command wizard. Happy editing!