Splunk Search
Highlighted

Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Builder

Hi,

We would like to be able to search a log file for a certain pattern or string, and then be able to return neighbouring events (say, 5 lines before or after the matched event. however we specify). This would be useful in a case where we want to see if a particular event is caused by, or causes any other events.

This functionality would be similar to using the -A (--after-context) and -B (--before-context) options of GNU grep.

Is this possible in Splunk?

Cheers,

Glenn

Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Splunk Employee
Splunk Employee

There is a doc on how to do this in our 3.x version. You can use the exact same technique in 4.x, but a search command like localize might be better suited:

... | localize maxpause=5m | map search="search failure starttimeu=$starttime$ endtimeu=$endtime$"

http://www.splunk.com/base/Documentation/latest/SearchReference/Localize

Here is a sample command from the wiki doc:

[search sourcetype="splunksource" splunk_event | stats min(_time) as eventstarttime | eval starttimeu=eventstarttime-900 | eval endtimeu=eventstarttime+900 | fields + starttimeu endtimeu]

http://www.splunk.com/wiki/Community:FindingSurroundingEvents

Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Builder

Either I'm doing it wrong or this doesn't give me what I want.

host=splunkhost source=splunksource "search_text" | localize maxpause=5m | map search="search starttimeu=$starttime$ endtimeu=$endtime$"

Gives:
time count density duration endtime starttime

1 18/05/2010 09:25:06.000 1 -1.000000 0 1274171136 1274171076

IE. not other events near the searched one

  • [search host="bruatosd001*" source=/var/log/messages spam | eval starttimeu=time | eval endtimeu=time+300 | fields + starttimeu endtimeu]

just gives: Error in 'UnifiedSearch': Unable to parse the 'Missing LHS for AND' search.

0 Karma
Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Builder

I should say that I'm using v4.1.2. Also, sorry that the above has completely lost its formatting. The "time count density..." bit is actually supposed to be a table, as returned by the search passed to localize. This came up in the "Results table" view section, there were no actual event results in the "Events list" view section.

0 Karma
Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Engager

Glenn, I'm in the same boat as you and am honestly extremely surprised at how hard this task is. When I started looking this morning, I thought for certain I'd find a command or modifier to let a search return context around a found message.

localize + map seems completely useless. Sure, it tells you other events happened nearby but there's no way to actually see the events. I'm not sure how anyone can claim that this is an alternative to using a subsearch to see surrounding events (unless there's some trick we're missing, in which case it should be documented).

As for subsearches, I think this might be a bug. We're using a trial (4.0.9) and I get the exact same behavior as you're describing with the "Missing LHS for AND" error. However, I only get that if I try to have the fields command spit out the start and end times (tried using 'earliest' and 'latest' to no avail as well).

I'm experimenting and maybe I can trick it. It's on my list for the next time I talk to our rep as well, because focused queries that show the context are extremely important.

Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Explorer

I too would like to be able to search the logs with the results either showing the context (surrounding events) or without actually filtering out anything. This doesn't seem possible at first glance though. Anyone have any ideas?

0 Karma
Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Builder

This is still unanswered (not sure why the below answer has at least two upvotes - its wrong), so I'm going to start a bounty. I often have developers asking me for this, plus there are others in this thread that are interested... there must be a way? If it's not answered after this, I'll be raising an enhancement request.

0 Karma
Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Motivator

Glenn, I think Simeon was right trying to use map and localize. I tried to do this as well, but as you found out once a search (saved or otherwise) is passed to map it doesn't spit out any results. This might actually be a defect in the product, I'll log a case about it.

Here is however how you can work around it. It's not as pretty as map, but it works. First the search:

* [search index=foo sourcetype="WindowsUpdateLog" Synchronizing | eval search="index=\"" + index + "\" host=\"" + host + "\" sourcetype=\"" + sourcetype + "\"" | eval earliest=min(_time-30) | eval latest=max(_time+30) | fields + search earliest latest | format "" "(" "" ") OR" "" "_cd=0"]

easier to read:

* [search index=foo sourcetype="WindowsUpdateLog" Synchronizing 
| eval search="index=\"" + index + "\" host=\"" + host + "\" sourcetype=\"" + sourcetype + "\"" 
| eval earliest=min(_time-30) 
| eval latest=max(_time+30) 
| fields + search earliest latest 
| format "" "(" "" ") OR" "" "_cd=0"]

This search will return all events within 30 seconds of the events you are searching for.

I am modifying the format command to spit out search ranges that we are passing to the outer wildcard search. If I don't try to modify the output of the format command I get the same Missing LHS for AND search error you were receiving, and if I just strip the outer parenthesis a trailing OR is added. I couldn't find a different way around the trailing OR so I decided to just give it a bogus event index id value (_cd=0). If there is a better way around it I'd appreciate if someone could leave me a comment.

Here is some sample output of the subsearch:

( earliest="1278605705.287" latest="1278605765.287" index="foo"  host="host1"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278605704.974" latest="1278605764.974" index="foo"  host="host1"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278605173.017" latest="1278605233.017" index="foo"  host="host2"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278605171.423" latest="1278605231.423" index="foo"  host="host2"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278602104.633" latest="1278602164.633" index="foo"  host="host1"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278602104.242" latest="1278602164.242" index="foo"  host="host1"  sourcetype="WindowsUpdateLog" ) OR _cd=0

I hope this will work for you.

Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Builder

Thanks. This also works, but I must say, its particularly complicated, and would scare my users off Splunk! It's a little less flexible than Ledion Bitincka's transaction solution, as transaction can handle returning +/- a number of events as well as +/- time, but has the benefit of automatically restricting the search to a single log file (you dont have to know where the event is beforehand), and returning events as separate events - not one large transaction block. It will be difficult to decide who to award the bounty to.

0 Karma
Highlighted

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Motivator

@Glenn, you could also add this as a custom workflow action and have a user kick this off with a click of a button. They would not need to remember the complicated syntax at that point.

0 Karma