Splunk Search
Highlighted

Search for Events that Happened Around the time of Other Events that Also meet other Criteria

Path Finder

How do I make a query that will search for events that happened around the same time as the results of another query?

So for example, if I search for "host='foo' bar", how can I find other events that happened within say 5 seconds of any "host='foo' bar" events?

Secondly, how would I modify that search so it returns events that happened within 5 seconds of "host='foo' bar" events that also match a query such as "baz"? So only events within 5 seconds of "host='foo' bar" that have the string "baz" in them (while still including the "host='foo' bar" events.

Hope I am being clear, these are basically a sort of event correlation searches and I imagine are pretty basic, I just can't find the magic Google keywords.

Tags (2)
Highlighted

Re: Search for Events that Happened Around the time of Other Events that Also meet other Criteria

Builder

I think I understand what you're asking, by default splunk does something a little different with events "time", run your search "host='foo' bar" then after the events are listed, simply click the time on the event line you'd like to search around (the date next to the gray pull-down menu in the actual event line). This will then set the search time to the specified time you clicked on. Splunk I believe has a 30 second default window for you to "drill" down on. After your search time is constrained, then simply type back in your search line the new search you'd like to look for, such as "baz"

Hope this helps!

Highlighted

Re: Search for Events that Happened Around the time of Other Events that Also meet other Criteria

Path Finder

I did find that, but that isn't really ideal for seeing patterns. Always better to have it in front of you. But while we are on that, can you manually increase that 30 second windows somehow?

0 Karma
Highlighted

Re: Search for Events that Happened Around the time of Other Events that Also meet other Criteria

Builder

After you zoom in to a selected time frame, there is a "zoom out" button that tries to intelligently expand a time frame. Try giving that a shot!

0 Karma
Highlighted

Re: Search for Events that Happened Around the time of Other Events that Also meet other Criteria

Splunk Employee
Splunk Employee

I found a much better method
using the wonderful FORMAT default option.

  • the main search is your second search
  • the sub search is your first search to find the event time and the other parameters the time will be used to define a time range of 120 seconds around each events. the results are formatted as OR/AND conditions using FORMAT.

Example to find the ip of a server with ERROR from an index into another index, 2 minutes around.


sourcetype=myothersourcetype [ search ERROR
| eval earliest=time-120 | eval latest=time+120 | fields src_ip earliest latest | FORMAT "(" "(" "" ")" "OR" ")" ]

Here is a simple search to find events 120 seconds around a splunk restart


index=internal [ search index=internal sourcetype="splunkd" "(build"
| eval earliest=time-120 | eval latest=time+120 | fields host earliest latest | FORMAT "(" "(" "" ")" "OR" ")" ]

Highlighted

Re: Search for Events that Happened Around the time of Other Events that Also meet other Criteria

Splunk Employee
Splunk Employee

If you are looking for the classic : show me all events 10 minutes around midnight last night, you can use this version.

index=_internal [|stats count| eval _time=now() | bucket _time span=1d
| eval earliest=_time-60 | eval latest=_time+60 | table earliest latest | FORMAT "(" "(" "" ")" "OR" ")"]

0 Karma
Highlighted

Re: Search for Events that Happened Around the time of Other Events that Also meet other Criteria

Communicator

This may not be exactly what you're after, but it's worth mentioning that the "Power Actions" app (http://apps.splunk.com/app/1547/) implements some of this functionality in the workflow - you can search +- 1 minute or +- 2 seconds around any event. That's not as robust as what you can do with yannK's approach, of course, but it's easy to implement for people who don't know the Splunk search language.

0 Karma