Splunk Search

I need to alert when one value from last 24 hours multiplied by 2 differs from dedup of 2 fields from the past 60 minutes

dorgra
Path Finder

I have 36 servers that forward event sources with 2 distinct values. I need to compare the number of system names (from previous 24 hours) * 2 and compare that result to a dedup of system_name, and event_source (from past 60 minutes). If the results don't match, I need an alert. One more problem is, the number of system names is subject to change.

index=myIndex sourcetype=mySourcetype 
    [ search index=myIndex sourcetype=mySourcetype earliest=-1d
    | dedup system_name
    | stats count(eval(system_name*2)) as SysNmCount ]
    [ search index=myIndex sourcetype=mySourcetype earliest=-60m
    | dedup event_source, system_name
    | stats count as EvntCount ]
| stats count(eval(SysNmCount/EvntCount)) as FinalCount

This search returns 0. I was thinking I could use a

| Where 

clause to alert when the FinalCount doesn't equal 1 when I get the return I need. I would like to include the

dedup event_source, system_name
in the alert email so the system operators can start looking for the missing combination(s) problem.

The search that works right now has the value of 72, but it changes from time to time.
index=myIndex sourcetype=mySourcetype

| dedup event_source, system_name
| eventstats count as total 
| where (total < 72 OR total > 72)
| fields system_name event_source

Any suggestion would be appreciated.

0 Karma

dorgra
Path Finder

See final comment for the solution that works.

0 Karma

dorgra
Path Finder

Researching got me a little further. Since I have 2 subsearches, I use "set union". Now I need a table that is created ONLY when SysNmCount and EvntCount are different.

Any suggestions?

| set union
[search index=myIndex sourcetype=mySourcetype earliest=-60m@m
| dedup system_name event_source
| stats count as EvntCount]
[search index=myIndex sourcetype=mySource earliest=-24h
| dedup system_name
| stats count as total
| eval SysNmCount = total * 2]
| table SysNmCount EvntCount

0 Karma

dorgra
Path Finder

I finally figured it out. Instead of "diff" or "union" or "makeresults" there is a simple solution. Sorry, I'm still new to SPL.

index=myIndex sourcetype=mySourcetype earliest=-24h NOT
[search index=myIndex sourcetype=mySourcetype earliest=-1h
| dedup system_name event_source
| fields system_name event_source
| table system_name, event_source]
| dedup system_name event_source
| fields system_name event_source
| table system_name, event_source
| eventstats count as total
| where total > 0

Setting this to run every hour (at 8 minutes past for resource issues) as an alert returns the event sets that are there for 24 hours but not the previous hour.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...