topic Re: Search and display surrounding context in Splunk Search

Search and display surrounding context

d3vino — Thu, 15 Sep 2011 19:44:58 GMT

Hi,

I am able to perform a search of some logs, but I would like to see the context surrounding a specific event.

For example, I run a search for "foo" and receive many results, but I would like to see the lines immediatly preceding the line with "foo" in it.

Using a linux grep command, I could run "grep -B5 foo /var/log/messages" to find lines with "foo" in them and the 5 lines preceding it. Is there a way in Splunk to perform the same type of search?

Re: Search and display surrounding context

acdevlin — Thu, 15 Sep 2011 20:44:20 GMT

The transaction command sounds like what we want here: http://docs.splunk.com/Documentation/Splunk/4.2.3/SearchReference/Transaction

I'd try something like this.

... | transaction endswith="foo" maxevents=6

Re: Search and display surrounding context

l0pher — Tue, 21 Oct 2014 21:51:04 GMT

I'm not sure if this is what the OP wanted. They just want to extract the surrounding events for an event matching a criteria. the endswith field will only query for events with "foo", but the need is to just extract any and all events surrounding the matched event.

Re: Search and display surrounding context

eyecantell — Thu, 23 Apr 2015 19:13:41 GMT

I would love to see this answer expanded as well. I have tried the following to return events that are not of category 'INFO' and are within one second before an 'Error' but its not quite right...

message="Error" | localize timebefore=1s| map search="search category!=\"INFO\" starttimeu=$starttime$ endtimeu=$endtime$" | transaction maxspan=1s

Re: Search and display surrounding context

eyecantell — Fri, 24 Apr 2015 00:46:54 GMT

I think what we are looking for here is a combination of eval and map. Use eval to set up a time window and map to iterate over the original search results - grabbing the events surrounding them.

To get events that happened surrounding the original set of events (to gain the desired context) we can use the following to give all events from 2.5 seconds before to 1.5 seconds after the original set:

message="Error" |eval mystarttime=_time-2.5 | eval myendtime=_time+1.5 | map search="search _time<$myendtime$ _time>$mystarttime$"

Note that this may take a bit, and beware that you can get events listed more than once if the time window you set up overlaps more than one of the original events.

If you want map to do more than the default limit of 10 searches, you will want to add the maxsearches option

map search="search _time<$myendtime$ _time>$mystarttime$" maxsearches=99

You can also make the results more intuitive for reading with the transaction command:

message="Error" |eval mystarttime=_time-2.5 | eval myendtime=_time+1.5 | map search="search _time<$myendtime$ _time>$mystarttime$" maxsearches=99 | transaction maxspan=4s

it will group each set of events into a single transaction on the return. In this case we are saying group any events within 4 seconds of each other.

I would love to have a more efficient way of doing this search. Unfortunately localize is not vey intuitive and is restricted to whole second increments, and the startimeu and endtimeu time functions for search didn't seem to give any better results.

Here is some very basic background info in case you are new to map and eval:

The map command runs a new search for each of the events passed to it. Its a typical looping operator.

For example, if

search message="Error"

returns 5 results, then

search message="Error" | map search="search message=\"$message$\"

should return 25 results because it performs a search for the same message again as it loops over each of the original results (5x5=25). Indeed it does.

We could return the original set of results by limiting them to the time they were encountered (assuming none of them happened at once, in which case there would still be duplicates on the corresponding times)

message="Error" | eval mytime=_time | map search="search message=$message$ _time=$mytime$"

The eval command gives us the ability to set up new variables based on those found in the previous result(s)

Good luck!

Re: Search and display surrounding context

dasveruckte — Mon, 03 Oct 2016 15:55:40 GMT

Had similar question. Found answer in this article

https://answers.splunk.com/answers/122983/what-is-the-easiest-way-to-find-log-lines-preceding-an-error.html