topic Re: How do I get a large count of events over a period of time? in Splunk Search

How do I get a large count of events over a period of time?

eli_mz — Tue, 29 Sep 2020 13:53:14 GMT

I'm trying to write a search string that will count firewall events up to 900k over 60 minutes to trigger an alarm when the event count goes under the 900k events. However, after reviewing the job using the search string below with the time range set in the drop down, I noticed that the search job scans 931k events before reaching the 900k count. Aside from the 31k additional events returned, the search takes a while to run (1-2 min).

I've also toyed with the metadata command to calculate traffic flow which runs significantly faster, but I'm not familiar with it enough to know if it's a viable solution. Any help or guidance would be appreciated.

Re: How do I get a large count of events over a period of time?

maraman_splunk — Sat, 29 Apr 2017 14:08:21 GMT

try something like that (use the metadata, will be fast)

| tstats count where index=myindex sourcetype=mysourcetype by _time span=60m | where count <900000

Re: How do I get a large count of events over a period of time?

rjthibod — Sat, 29 Apr 2017 17:11:42 GMT

First, your search will run much faster if you use tstats.

| tstats count where index=some_index sourcetype=some_stype

What is unclear though is the best way to use it, because I a not quite understanding what output you want. Do you just want to trigger an alert via a saved search when the count is under 900K? Are you going to limit the search to earliest=-60m or do you plan to search over a longer period and want to look in 60 minute buckets?

Re: How do I get a large count of events over a period of time?

mattymo — Sat, 29 Apr 2017 17:19:34 GMT

definitely go with tstats, also, what is the goal of counting up to 900k but alarming under??

Seems like a strange threshold for firewall traffic alarming...

Re: How do I get a large count of events over a period of time?

eli_mz — Mon, 01 May 2017 14:09:15 GMT

The search is going to be limited to a 60 minute bucket. I'm planning on running this from a saved search.

The idea is to trigger an alert when the events created\forwarded fall below 900k events as an indicator of potential issues. I've had instances where the box is up and running but the events are not being forwarded.

Re: How do I get a large count of events over a period of time?

eli_mz — Mon, 01 May 2017 14:09:19 GMT

The idea is to trigger an alert when the events created\forwarded fall below 900k events as an indicator of potential issues. I've had instances where the box is up and running and seemingly running normally (no network issues) but the events are not being forwarded. Looking at one of those events overtime I can see the events coming in at a decreasing rate before it finally reaches "0".

Re: How do I get a large count of events over a period of time?

rjthibod — Mon, 01 May 2017 14:12:47 GMT

Then this saved search should work. You will need to setup the alert conditions to look for any results (i.e., resultCount > 0).

| tstats count where index=some_index sourcetype=some_stype | where count < 900000

Re: How do I get a large count of events over a period of time?

eli_mz — Mon, 01 May 2017 15:12:12 GMT

Yup, this will work for my purpose. Off to do a bit more reading. Thanks!

Re: How do I get a large count of events over a period of time?

eli_mz — Mon, 01 May 2017 15:14:19 GMT

Excellent. This will work for my purpose.

Both rjhibod and mmodestino pointed me to that as well. Thank you all.

Re: How do I get a large count of events over a period of time?

DalJeanis — Mon, 01 May 2017 15:31:57 GMT

The by _time span=60m will break up all of history into 60 minute chunks and report if ANY of them are below 900K.

Just use earliest=-1h.