topic Re: Searching the latest event quickly in Splunk Search

Searching the latest event quickly

laberthelemy — Tue, 20 Sep 2016 09:48:38 GMT

Hello

I would like to check if my firewall rules are used or not.
For that, I'm doing something like that :
index=firewall TERM(src=10.0.0.1) OR TERM(src=10.0.0.2) TERM(dst=10.0.0.3) TERM(port=12345) | stats latest(_time)

But it tooks a long time (on all events) ... However, it should not check buckets in the past, when it finds events 1 hour ago ...

Do you have a tip for that ?

Re: Searching the latest event quickly

echalex — Tue, 20 Sep 2016 12:31:41 GMT

Hi,

If I understand correctly, your main concern is slowness? Yes, if you just need the last event, you can use head N, which will give you the N latest events.

Example:

index=firewall TERM(src=10.0.0.1) OR TERM(src=10.0.0.2) TERM(dst=10.0.0.3) TERM(port=12345) | head 1

HTH

Re: Searching the latest event quickly

richgalloway — Tue, 20 Sep 2016 12:53:23 GMT

If you only need the last hour then:

index=firewall TERM(src=10.0.0.1) OR TERM(src=10.0.0.2) TERM(dst=10.0.0.3) TERM(port=12345)  earliest=-1h | stats latest(_time)

Re: Searching the latest event quickly

laberthelemy — Tue, 20 Sep 2016 13:03:09 GMT

Thanks for your answers, but it's still long ...
In fact, I don't know when is the last event, so I must search in "All time"
And It seems that head 1 is evaluated after the full search.

Re: Searching the latest event quickly

echalex — Tue, 20 Sep 2016 14:39:07 GMT

If you don't know when the last event is, then head will help you. Apparently your milage varies quite a bit, but for me head seems to terminate the search very efficiently. I'm also searching over a "All time" in a huge index, but with head it ends within a second.
Did you see that you need to use head before you use stats. In fact, I left out stats completely in my answer.

Re: Searching the latest event quickly

echalex — Tue, 20 Sep 2016 14:39:53 GMT

In your example, if you need stats, you would want to do this:

index=firewall TERM(src=10.0.0.1) OR TERM(src=10.0.0.2) TERM(dst=10.0.0.3) TERM(port=12345) | head 1 |stats latest(_time)

Re: Searching the latest event quickly

echalex — Tue, 20 Sep 2016 14:40:41 GMT

See the comment I added to my original answer/

Re: Searching the latest event quickly

laberthelemy — Wed, 21 Sep 2016 08:51:54 GMT

| head 1 | stats latest(_time) is really efficient
Thank you so much (even if I don't understand why we need head 1 🙂

Re: Searching the latest event quickly

laberthelemy — Wed, 21 Sep 2016 08:55:10 GMT

I understand now : it was because I was searching on sourcetype instead of index. Sorry ...

Re: Searching the latest event quickly

echalex — Wed, 21 Sep 2016 09:31:39 GMT

Well, head restricts the number of results to the number you specify and Splunk discards the rest, so after receiving say 100 events, Splunk knows it doesn't have to look for more. head shows the newest N events it encounters. (In the specified time frame.) There is also a tail command that will give you the oldest N events. The tail command is therefore slower, since it has to churn through the entire result set first.