Alerting

How to alert when particular field value is changed

svelagala
Loves-to-Learn

I have an index with certain field values. I want to be notified when specific field value changes, I am aware of using stream stats change_on_reset=true by that field name.

But the challenge here is I am not sure when that field has received last value. Suppose I have field X with 1,2,3 at Y time on Z day. Again, That field X got 4,5 values at P time on Q day. Here, I will not be aware of the day(Z) and time(Y) values to look back certain time range for that field values.

To summarize, If am aware  of time range to select like "last 7 days" or "last 24 hours", then I can see if that field value changed or not through stream stats. But here there is no definite time period flow of events in the index. Hence We are not sure when and what is the last event stored in that particular field. Can someone  help me on this requirement?

Labels (1)
0 Karma

dwaddle
SplunkTrust
SplunkTrust

Roughly .. use a lookup!  See https://www.splunk.com/en_us/blog/tips-and-tricks/maintaining-state-of-the-union.html

 

Imagine it this way.  Say I run this search exactly one time, over all time.

index=foo sourcetype=bar 
| stats latest(somefield) as the_latest 
| outputlookup mylookup.csv

 

Once this is done, I have cached the "latest value" in a lookup.  I never have to go back over all time again to find the latest value.  I can now schedule a couple of searches to run over a smaller time window.  First to alert, then to update the latest value in the lookup.  The alert search looks something like this, running every (say) 15 minutes:

index=foo sourcetype=bar
| stats latest(somefield) as current_value
| inputlookup append=true mylookup.csv
| stats first(current_value) as current_value, first(the_latest) as the_latest
| where current_value != the_latest

 

Now, I have to account for the fact that I will need to update my lookup occasionally.  I can rerun the first search, over a smaller time window (1 hour?) in order to keep my cached value in my lookup.

 

A really clever approach would be to add the necessary eval and outputlookup statements to your alerting search such that, if the value changes we update the lookup then.  I've not tested it, but maybe something like

index=foo sourcetype=bar
| stats latest(somefield) as current_value
| inputlookup append=true mylookup.csv
| stats first(current_value) as current_value, first(the_latest) as the_latest
| where current_value != the_latest
| eval the_latest=if(current_value != the_latest, current_value, the_latest)
| fields the_latest
| outputlookup mylookup.csv

 

I am concerned though about trying to do this in an alerting search, I'm not sure what the output of the alert will look like.

0 Karma

svelagala
Loves-to-Learn

Thanks @dwaddle for your time, this looks promising(haven't tried yet). But I think you missed considering comparison of all the values in the latest log/transaction.

That means there might be multiple values in the same transaction. Suppose X field has 1,1 and 2 as values at Y time range. We are comparing first(X) i.e., 1 with latest(lookup) which is good but we still need to compare all the values in this transaction i.e., 1 with 1,2 values and alert us since 2 is a different value.

Hope you understood, Can you help me with this?

0 Karma

dwaddle
SplunkTrust
SplunkTrust

I don't actually follow what you're saying, sorry.  You may be on your own for figuring out the exact approach here.  But, generally speaking, lookups are one of the most effective ways to cache information from searches for use over arbitrary time windows.

0 Karma

rnowitzki
Builder

The alert will have a schedule, so it runs every hour, day, week etc.

So,  the timeframe of the search just has to cover that range + maybe some buffer and it should catch every change. 

So, when you have your alert set to run every hour, you look back something like 65 minutes in your search. That way you shouldn't miss a change to the field values. If you run it daily, you maybe want to look back 25 hours. Not even sure if that buffer is needed. It should work with the exact same range/timeframe as you set the alert to.

Maybe I miss something/missunderstood you.

--
Karma and/or Solution tagging appreciated.
0 Karma

svelagala
Loves-to-Learn

Thanks @rnowitzki  for your response. I get your point but unfortunately, logs of that index would be of indefinite time period. That means Suppose field got last value in last 2 weeks, next log might be earlier(like within 1 week) than that or later(may be after 25 days) as well. Hence, How to know the time range to be selected to compare recent/current value of that field with the lastly stored value in that field.

To give more picture on the requirement I am looking for a query that will help in alerting whenever a particular field value changes by comparing the 1st value(suppose 3 values are there) of recent log/transaction with the last/old value of that field (no certain idea of when the last log came in) and also comparing within these 3 values of recent transaction if there is any new value within them.

Hope you got my concern.

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Hi Splunky people! We are excited to share the newest updates in Splunk Enterprise 9.3!Admins and Analyst can ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...