Log alerts - Alert that tell when a log source stops sending logs to Splunk

[BTB]

I'm trying to build an alert that looks at the number of logs from the past three days and then compares it to the number of logs from the previous three days before that. I want to set off an alert if the log numbers drop by 30% or more in any period. I've seen this done with search and with _index, but I'm unsure which way is best. I don't want to build almost 100 searches for 100 different source types, and I'd much rather do it by the twenty-something indexes. I'm not sure if ML is the right way to do this but I've seen times when logs stop flowing and it isn't noticed for days and I want to prevent that from happening. Any help is appreciated. 

[jotne]

For a more robust solution have a look at the TrackMe app.
Its free if you are not a big user of it.  Can be som work to set it up.

https://splunkbase.splunk.com/app/4621

[gcusello]

Hi @BTB,

let me understand: you want to check if you received in the last three days less than 30% of the logs that you received in the previous 3 days for each sourcetype, is it correct?

If this is your requirement, you could try something like this:

| tstats 
   count
   latest(_time) AS _time
WHERE index=* earliest=-6d
BY sourcetype
| eval period=if(_time>now()-86400*3,"Last","Previous"
| stats 
   sum(eval(if(period="Last",count,0))) AS Last
   sum(eval(if(period="Previous",count,0))) AS Previous
   dc(period) AS period_count
   values(period) AS period
   BY sourcetype
| eval diff_perc=(Last-Previous)/Previous*100
| where diff_perc<30

if you want a different algorithm you can implement using my approach.

Ciao.

Giuseppe

[BTB]

Looks closer 

[BTB]

[gcusello]

Hi @BTB,

at first, move the condition index=<my_index in the where condition not in the beginning of the search.

Anyway, in my answer I missed a parenthesys in the sixth row, use this:

| tstats 
   count
   latest(_time) AS _time
WHERE index=<my_index> earliest=-6d
BY sourcetype
| eval period=if(_time>now()-86400*3,"Last","Previous")
| stats 
   sum(eval(if(period="Last",count,0))) AS Last
   sum(eval(if(period="Previous",count,0))) AS Previous
   dc(period) AS period_count
   values(period) AS period
   BY sourcetype
| eval diff_perc=(Last-Previous)/Previous*100
| where diff_perc<30

 Ciao.

Giuseppe

[BTB]

We are closer but not there yet. 

[gcusello]

Hi @BTB ,

yes, as @yuanliu said, <my_index> means that you have to use the name of your index, without the angular brackets, and it should work.

Ciao.

Giuseppe

[yuanliu]

I think you take <my_index> in @gcusello's solution a step too literally.  The angular brackets (<>) are not part of search string, just a suggestion for you to substitute with your own index name.  If I guess correctly, your index name is "ops_sec", not "<ops_sec>".  Is it?

The solution should work.  Here is an alternative that takes advantage of some tstats options to remove most subsequent calculations.

| tstats count WHERE index = ops_sec ``` replace ops_sec with your index name ```
  earliest=-6d
  BY sourcetype _time span=3d
| stats list(count) as count3d by sourcetype
| eval diff_perc = (tonumber(mvindex(count3d, 1)) - tonumber(mvindex(count3d, 0))) / tonumber(mvindex(count3d, 0)) * 100
| where diff_perc < 30

This is a simulation using internal indexes on my laptop:

| tstats count WHERE index = _* ``` replace _* with your index name ```
  earliest=-6d
  BY sourcetype _time span=3d
| stats list(count) as count3d by sourcetype
| eval diff_perc = (tonumber(mvindex(count3d, 1)) - tonumber(mvindex(count3d, 0))) / tonumber(mvindex(count3d, 0)) * 100
| where diff_perc < 30

Output is

sourcetype	count3d	diff_perc
mongod	183 3	-98.36065573770492
splunk_telemetry	13 8	-38.46153846153847
splunk_web_access	17 5	-70.58823529411765
splunk_web_service	274 17	-93.7956204379562

[BTB]

Nice! Exactly what I was looking for! I did know to replace with the index. Opsec was just "Operations Security" to not reveal what we are running. 😀