Splunk Search
Highlighted

I need to alert when one value from last 24 hours multiplied by 2 differs from dedup of 2 fields from the past 60 minutes

Path Finder

I have 36 servers that forward event sources with 2 distinct values. I need to compare the number of system names (from previous 24 hours) * 2 and compare that result to a dedup of systemname, and eventsource (from past 60 minutes). If the results don't match, I need an alert. One more problem is, the number of system names is subject to change.

index=myIndex sourcetype=mySourcetype 
    [ search index=myIndex sourcetype=mySourcetype earliest=-1d
    | dedup system_name
    | stats count(eval(system_name*2)) as SysNmCount ]
    [ search index=myIndex sourcetype=mySourcetype earliest=-60m
    | dedup event_source, system_name
    | stats count as EvntCount ]
| stats count(eval(SysNmCount/EvntCount)) as FinalCount

This search returns 0. I was thinking I could use a

| Where 

clause to alert when the FinalCount doesn't equal 1 when I get the return I need. I would like to include the

dedup eventsource, systemname
in the alert email so the system operators can start looking for the missing combination(s) problem.

The search that works right now has the value of 72, but it changes from time to time.
index=myIndex sourcetype=mySourcetype

| dedup event_source, system_name
| eventstats count as total 
| where (total < 72 OR total > 72)
| fields system_name event_source

Any suggestion would be appreciated.

0 Karma
Highlighted

Re: I need to alert when one value from last 24 hours multiplied by 2 differs from dedup of 2 fields from the past 60 minutes

Path Finder

Researching got me a little further. Since I have 2 subsearches, I use "set union". Now I need a table that is created ONLY when SysNmCount and EvntCount are different.

Any suggestions?

| set union
[search index=myIndex sourcetype=mySourcetype earliest=-60m@m
| dedup systemname eventsource
| stats count as EvntCount]
[search index=myIndex sourcetype=mySource earliest=-24h
| dedup system_name
| stats count as total
| eval SysNmCount = total * 2]
| table SysNmCount EvntCount

0 Karma
Highlighted

Re: I need to alert when one value from last 24 hours multiplied by 2 differs from dedup of 2 fields from the past 60 minutes

Path Finder

I finally figured it out. Instead of "diff" or "union" or "makeresults" there is a simple solution. Sorry, I'm still new to SPL.

index=myIndex sourcetype=mySourcetype earliest=-24h NOT
[search index=myIndex sourcetype=mySourcetype earliest=-1h
| dedup systemname eventsource
| fields systemname eventsource
| table systemname, eventsource]
| dedup systemname eventsource
| fields systemname eventsource
| table systemname, eventsource
| eventstats count as total
| where total > 0

Setting this to run every hour (at 8 minutes past for resource issues) as an alert returns the event sets that are there for 24 hours but not the previous hour.

0 Karma
Highlighted

Re: I need to alert when one value from last 24 hours multiplied by 2 differs from dedup of 2 fields from the past 60 minutes

Path Finder

See final comment for the solution that works.

0 Karma