Navigating through Text with `grep`, `cut`, and `awk`

While working with text files in Unix-like systems, three commands come in very handy for processing and analyzing data: grep, cut, and awk. These powerful utilities can be combined in countless ways to filter, rearrange, and manipulate text. Let’s dig into some fascinating ways to use them, both individually and together.

1. Chaining grep and cut

grep is used to search text, while cut is used to slice it. We can use them in tandem to refine our search and extract only the information we need. Here’s an example:

grep SEARCHTERM file.txt | grep "NEWSSEARCHTERM" | cut -f1,17- -d" "

This command first uses grep to find lines containing SEARCHTERM. From these lines, it then finds lines containing NEWSSEARCHTERM. Finally, cut is used to extract the 1st and 17th (and onward) fields from these lines, using a space as a delimiter.

And here’s a real-life example:

grep ' 500 ' FILENAME_log_2014-09-11 | grep -v '200 -' | cut -f1,18- -d' '

This reads FILENAME_log_2014-09-11, finds status 500, excludes 200 -, and extracts the 1st and 18th onward fields. cut -d' ' treats each single space as a delimiter, so repeated spaces create empty fields. For whitespace-separated columns I use awk, which collapses runs of whitespace:

awk '$0 ~ / 500 / && $0 !~ /200 -/ { print $1, $18 }' FILENAME_log_2014-09-11

2. grep Flags and Options

grep offers many useful flags. GNU grep’s -P enables Perl-compatible regular expressions; it is not portable to every grep. -A and -B display lines after and before the match, while -C displays both:

grep -P '\[\d+\] (Publish Event:|Sent Points:|Sending Points:|Point received)?' file.log
grep -A 5 -B 3 'Hello' file.log
grep -C 3 'Hello' file.log

3. Live Monitoring with tail, grep, and cut

When monitoring log files, you can use tail -f to follow the file’s updates in real time. This can be piped into grep and cut for live filtering and slicing:

tail -F /var/log/server.log | grep --line-buffered -i 'no configuration found' | cut -d' ' -f8-11,13

-F follows the file name across common log rotations. The separate grep, cut, and awk examples remain useful for learning each stage.

4. grep OR and NOT Operators

grep supports logical OR (|) and NOT (-v) operations, which can be combined with -E for extended regular expressions:

grep 'pattern1\|pattern2' filename
grep -E 'pattern1|pattern2' filename
grep -v 'pattern1' filename

5. Using awk

awk is a powerful text processing language that can be used for tasks such as scanning for patterns and performing actions on matched lines. Here’s an example of awk being used to print the third column of a file:

awk '{print $3}' filename

You can also use awk in combination with grep to filter lines before processing them. Here’s an example where we first filter lines containing “Error” and then print the fifth column:

grep "Error" filename | awk '{print $5}'

In conclusion, grep, cut, and awk provide a wide array of functionalities for text processing in Unix-like systems. Mastering these commands and understanding how to combine them will greatly enhance your ability to handle and analyze text data.