Splunk Search

How do you use multiple thresholds from a timechart for a single alert for an arbitrary number of devices?

michaelsplunk1
Path Finder
_timedevice1_avgdevice2_avgdevice3_avgdevice4_avg
2022-04-07 00:003431122
2022-04-07 01:0021764187
2022-04-07 02:002183232
2022-04-07 03:001233654
2022-04-07 04:00782143
2022-04-07 05:001131721
2022-04-07 06:0019121916
2022-04-07 07:0015101219
2022-04-07 08:0042196

 

I have a table of averages for an arbitrary number of arbitrary devices as shown above. How do I use these averages as thresholds for alerts about these devices? I'm trying to have a search that runs every 15 minutes to check which devices have exceeded these averages.

For example, if a search is run at 06:45, and returns that device1 has a count of 10, device2 has a count of 15, device3 has a count of 21, and device 4 has a count of 2, send an alert that says device2 and device3 have exceeded their averages listed for the 06:00 hour (i.e., 12 and 19, respectively).

Labels (1)
Tags (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Here's an untested idea.  Round the runtime of the search to the beginning of the hour.  Look up the result in the _time field of the averages table, returning all device thresholds for that hour.  Compare the calculated average to the threshold and trigger an alert if the result count is not zero.

```Round the runtime of the search to the beginning of the hour. ```
| eval lookupTime=relative_time(_time, "@h")
```Look up the result in the _time field of the averages table, returning all device thresholds for that hour.  ```
| lookup averages.csv _time OUTPUT device1_avg AS device1_thresh, device2_avg AS device2_thresh, device3_avg AS device3_thresh, device4_avg AS device4_thresh
```Compare the calculated average to the threshold```
| where (device1_avg > device1_thresh OR device2_avg > device2_thresh OR device3_avg > device3_thresh OR device4_avg > device4_thresh)

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Here's an untested idea.  Round the runtime of the search to the beginning of the hour.  Look up the result in the _time field of the averages table, returning all device thresholds for that hour.  Compare the calculated average to the threshold and trigger an alert if the result count is not zero.

```Round the runtime of the search to the beginning of the hour. ```
| eval lookupTime=relative_time(_time, "@h")
```Look up the result in the _time field of the averages table, returning all device thresholds for that hour.  ```
| lookup averages.csv _time OUTPUT device1_avg AS device1_thresh, device2_avg AS device2_thresh, device3_avg AS device3_thresh, device4_avg AS device4_thresh
```Compare the calculated average to the threshold```
| where (device1_avg > device1_thresh OR device2_avg > device2_thresh OR device3_avg > device3_thresh OR device4_avg > device4_thresh)

 

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...