Manual log analysis with emacs
Created at 2015-11-28T18:07:58.000Z
Try out text-processing commands interactively
  1. Make new buffer with switch-to-buffer (C-x b) followed by arbitrary buffer name. Refs:
  1. Put a part of huge log files into the buffer by M-1 M-! with a shell command you like. Refs:
  1. Select a region in the buffer and try any commands on the region with C-| or C-u C-|.
Show streaming log

After you find out commands to process log, you can combine it with tail -f to show a log in a streaming way.

The good thing about tail -f is the command is not a pager command like less, so you can use it in emacs shell without trouble. Then, the new log will be shown in emacs buffer, which means you can the easily use previous technique like C-| or C-u C-| to process it interactively.

This is an example:

$ tail -n 100 -f log/development.log | awk -v skip=-1 '/GET.*(jpg|JPG|png|PNG|jpeg)/ { skip = 6 } skip-- >= 0 { next } 1'
Example explained: Awk, Grep and Sed

What I wanted to do with the previous example is cutting out image request from the whole log like this:

...
Started GET "/trip-planner/development/spot_photo/image/556960e169702d3e709c0700/thumb_9346382163_876b6cf2d6_o.jpg" for 127.0.0.1 at 2015-11-28 20:07:22 +0900
Processing by PagesController#home as JPEG
  Parameters: {"path"=>"trip-planner/development/spot_photo/image/556960e169702d3e709c0700/thumb_9346382163_876b6cf2d6_o"}
Redirected to https://trip-planner-production.s3.amazonaws.com/production/spot_photo/image/556960e169702d3e709c0700/thumb_9346382163_876b6cf2d6_o.jpg
Completed 302 Found in 47ms
...

Basically, grep cannot do this kind of hiding multiple lines after matching point as explained in stackoverflow grep's options like -A (after), -B (before) and -C (context) can be used only for showing some lines around matching point.

Actually, I don't understand how awk works yet. The awk command piped to tail -f in the previous example is just what I got from the same stackoverflow answer.

awk -v skip=-1 '/GET.*(jpg|JPG|png|PNG|jpeg)/ { skip = 6 } skip-- >= 0 { next } 1'

I think I like the below way by for and getline.

awk '/GET.*(jpg|JPG|png|PNG|jpeg)/ { for (i=1; i <= 6; i++) { getline } } 1'

sed realizes the similar thing with more intuitive (reasonable) syntax by specifing a range to hide (/<reg-exp0>/,/<reg-exp1>/) and delete command d:

sed -e "/GET.*jpg/,/Completed^^/d"
References