topic Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep? in Splunk Search

Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Glenn — Mon, 17 May 2010 21:57:39 GMT

Hi,

We would like to be able to search a log file for a certain pattern or string, and then be able to return neighbouring events (say, 5 lines before or after the matched event. however we specify). This would be useful in a case where we want to see if a particular event is caused by, or causes any other events.

This functionality would be similar to using the -A (--after-context) and -B (--before-context) options of GNU grep.

Is this possible in Splunk?

Cheers,

Glenn

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Simeon — Tue, 18 May 2010 00:18:42 GMT

There is a doc on how to do this in our 3.x version. You can use the exact same technique in 4.x, but a search command like localize might be better suited:

... | localize maxpause=5m | map search="search failure starttimeu=$starttime$ endtimeu=$endtime$"

http://www.splunk.com/base/Documentation/latest/SearchReference/Localize

Here is a sample command from the wiki doc:

[search sourcetype="splunksource" splunk_event | stats min(_time) as eventstarttime | eval starttimeu=eventstarttime-900 | eval endtimeu=eventstarttime+900 | fields + starttimeu endtimeu]

http://www.splunk.com/wiki/Community:FindingSurroundingEvents

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Glenn — Mon, 28 Sep 2020 09:12:33 GMT

Either I'm doing it wrong or this doesn't give me what I want.

host=splunkhost source=splunksource "search_text" | localize maxpause=5m | map search="search starttimeu=$starttime$ endtimeu=$endtime$"

Gives:
time count density duration endtime starttime

1 18/05/2010 09:25:06.000 1 -1.000000 0 1274171136 1274171076

IE. not other events near the searched one

[search host="bruatosd001*" source=/var/log/messages spam | eval starttimeu=_time | eval endtimeu=_time+300 | fields + starttimeu endtimeu]

just gives: Error in 'UnifiedSearch': Unable to parse the 'Missing LHS for AND' search.

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Glenn — Tue, 18 May 2010 17:30:37 GMT

I should say that I'm using v4.1.2. Also, sorry that the above has completely lost its formatting. The "time count density..." bit is actually supposed to be a table, as returned by the search passed to localize. This came up in the "Results table" view section, there were no actual event results in the "Events list" view section.

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

cjbehm — Sat, 26 Jun 2010 03:00:10 GMT

Glenn, I'm in the same boat as you and am honestly extremely surprised at how hard this task is. When I started looking this morning, I thought for certain I'd find a command or modifier to let a search return context around a found message.

localize + map seems completely useless. Sure, it tells you other events happened nearby but there's no way to actually see the events. I'm not sure how anyone can claim that this is an alternative to using a subsearch to see surrounding events (unless there's some trick we're missing, in which case it should be documented).

As for subsearches, I think this might be a bug. We're using a trial (4.0.9) and I get the exact same behavior as you're describing with the "Missing LHS for AND" error. However, I only get that if I try to have the fields command spit out the start and end times (tried using 'earliest' and 'latest' to no avail as well).

I'm experimenting and maybe I can trick it. It's on my list for the next time I talk to our rep as well, because focused queries that show the context are extremely important.

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

seren — Mon, 05 Jul 2010 00:47:10 GMT

I too would like to be able to search the logs with the results either showing the context (surrounding events) or without actually filtering out anything. This doesn't seem possible at first glance though. Anyone have any ideas?

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Glenn — Thu, 08 Jul 2010 16:53:30 GMT

This is still unanswered (not sure why the below answer has at least two upvotes - its wrong), so I'm going to start a bounty. I often have developers asking me for this, plus there are others in this thread that are interested... there must be a way? If it's not answered after this, I'll be raising an enhancement request.

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

ftk — Thu, 08 Jul 2010 23:19:10 GMT

Glenn, I think Simeon was right trying to use map and localize. I tried to do this as well, but as you found out once a search (saved or otherwise) is passed to map it doesn't spit out any results. This might actually be a defect in the product, I'll log a case about it.

Here is however how you can work around it. It's not as pretty as map, but it works. First the search:

* [search index=foo sourcetype="WindowsUpdateLog" Synchronizing | eval search="index=\"" + index + "\" host=\"" + host + "\" sourcetype=\"" + sourcetype + "\"" | eval earliest=min(_time-30) | eval latest=max(_time+30) | fields + search earliest latest | format "" "(" "" ") OR" "" "_cd=0"]

easier to read:

* [search index=foo sourcetype="WindowsUpdateLog" Synchronizing 
| eval search="index=\"" + index + "\" host=\"" + host + "\" sourcetype=\"" + sourcetype + "\"" 
| eval earliest=min(_time-30) 
| eval latest=max(_time+30) 
| fields + search earliest latest 
| format "" "(" "" ") OR" "" "_cd=0"]

This search will return all events within 30 seconds of the events you are searching for.

I am modifying the format command to spit out search ranges that we are passing to the outer wildcard search. If I don't try to modify the output of the format command I get the same Missing LHS for AND search error you were receiving, and if I just strip the outer parenthesis a trailing OR is added. I couldn't find a different way around the trailing OR so I decided to just give it a bogus event index id value (_cd=0). If there is a better way around it I'd appreciate if someone could leave me a comment.

Here is some sample output of the subsearch:

( earliest="1278605705.287" latest="1278605765.287" index="foo"  host="host1"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278605704.974" latest="1278605764.974" index="foo"  host="host1"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278605173.017" latest="1278605233.017" index="foo"  host="host2"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278605171.423" latest="1278605231.423" index="foo"  host="host2"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278602104.633" latest="1278602164.633" index="foo"  host="host1"  sourcetype="WindowsUpdateLog" ) OR ( earliest="1278602104.242" latest="1278602164.242" index="foo"  host="host1"  sourcetype="WindowsUpdateLog" ) OR _cd=0

I hope this will work for you.

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Ledion_Bitincka — Fri, 09 Jul 2010 01:24:51 GMT

you should be able to -A or -B (but not both) using the transaction command

equivalent of -B
.... | transaction endswith=(<search that matches the event of interest>) maxevents=<number of events in txn>

equivalent of -A
.... | transaction startswith=(<search that matches the event of interest>) maxevents=<number of events in txn>

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Glenn — Tue, 13 Jul 2010 19:16:49 GMT

This is a nice simple option, and does seem to partially work. Oddly, it only acts as expected when using the "endswith" transaction option. Then I get a single block of events, from just before my searched term (just what I want). However, when I replace "endswith" with "startswith" I get thousands of transaction events, that do not have the search term in them. Does this happen for you? My search for example (in case I have the syntax wrong):

sourcetype=LSWebMessaging | transaction startswith=("epic=DIAAAAACAA5J5AG") maxevents=10

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Glenn — Tue, 13 Jul 2010 19:20:29 GMT

Thanks. This also works, but I must say, its particularly complicated, and would scare my users off Splunk! It's a little less flexible than Ledion Bitincka's transaction solution, as transaction can handle returning +/- a number of events as well as +/- time, but has the benefit of automatically restricting the search to a single log file (you dont have to know where the event is beforehand), and returning events as separate events - not one large transaction block. It will be difficult to decide who to award the bounty to.

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

ftk — Tue, 13 Jul 2010 20:01:15 GMT

@Glenn, you could also add this as a custom workflow action and have a user kick this off with a click of a button. They would not need to remember the complicated syntax at that point.

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

Ledion_Bitincka — Thu, 22 Jul 2010 02:18:37 GMT

it seems like you've hit a bug !!! The current workaround would be to use another search after the transaction command to filter out "bad" transactions. So:
sourcetype=LSWebMessaging | transaction startswith=("epic=DIAAAAACAA5J5AG") maxevents=10 | search "epic=DIAAAAACAA5J5AG"

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

piebob — Thu, 17 Jul 2014 18:54:09 GMT

if your use case is to look for events around the time of a given event, this other Answer might be useful: http://answers.splunk.com/answers/9940/search-for-events-that-happened-around-the-time-of-other-events-that-also-meet-other-criteria

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

dlamas_splunk — Thu, 17 Jul 2014 19:09:32 GMT

This isn't exactly what you're asking for, but there is a UI workflow to narrow down the time window around a specific event to see what happened immediately before and after that event in 6.x

Click on the arrow next to the event you want to investigate in "List" view.
Scroll down to the time section of the fields listed underneath the event and click on the _time
You'll get a pop-up with some options to look at events before, after, or near this time. Choose what makes sense.
Search will re-run with your time range in the time picker. If the events you want to investigate are in a separate index or require more keywords/filter terms, you can change that and Splunk will continue looking within that time window until you change the time range in the time picker.

If you want to continue zooming out after this, you can use the zoom out controls on the green timeline.

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

icyfeverr — Tue, 23 Sep 2014 23:01:28 GMT

Hi ftk, I just stumbled upon this post and like your response. I saw where you were looking for how to get rid of the trailing OR, so thought I would respond a few years later :-), below is the way to do it.

| format "" "(" "" ")" "OR" ""

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

deepeshk79 — Thu, 02 Apr 2015 14:52:00 GMT

Hi - I tried the below but it's not working
OutOfMemoryError host=ojwerelto* | transaction startswith="OutOfMemoryError" maxevents=100

When I just search for OutOfMemoryError host=ojwerelto* I get five events, now for each event I want to display 100 lines after/before that event...

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

doksu — Wed, 15 Jul 2015 05:33:27 GMT

Event immediately before:
... | eval foo=if(searchmatch("something"),"true","false") | transaction startswith="foo=false" endswith="foo=true"

Event immediately after:
... | eval foo=if(searchmatch("something"),"true","false") | transaction startswith="foo=true" endswith="foo=false"

Re: Can Splunk filter/match events and bring back neighbouring events like GNU grep?

hrdwdmrbl — Tue, 05 Jun 2018 16:33:41 GMT

This is also very slow! I had to use sampling to get some results.
1. Can someone explain why its slow?
2. Any other suggestions to speed it up?