Alerting

How to create alert with customized limits for many agents?

exmuzzy
Explorer

I need to be alerted that the agent is exceeding its specified limit, and if the agent limit is not set, then the base limit triggered

To atchive that I have created such search string for alert

index=api | stats count as c by agent | search  (agent=nfc* c>700) OR (agent=mm_write c>5000) OR (agent=breeze c>1500) OR (agent=megafon_bitmap c > 1000) OR (agent=bankm_cashback с>5000)  OR (agent!=breeze agent!=nfc* agent!=mm_write agent!=bankm_cashback agent!=megafon_bitmap c>700)  

But it looks too cumbersome

Is there better way to do this?
I would like to maintain some kind of table of agents limits.

0 Karma
1 Solution

micahkemp
Champion

I think you'd be best served using a lookup table to define the limits, then filtering those where the count is greater than the limit with where. Example configuration below.

agent_limits.csv:

agent,limit
nfc*,700
mm_write,5000
breeze,1500
megafon_bitmap,1000
bankm_cashback,5000
*,7000

transforms.conf:

[agent_limits]
filename = agent_limits.csv
match_type = WILDCARD(agent)
max_matches = 1 

Run anywhere search demonstrating functionality:

| makeresults | eval agent="nfc1", count=650
| append [| makeresults | eval agent="nfc2", count=750]
| append [| makeresults | eval agent="breeze", count=5001]
| lookup agent_limits agent OUTPUT limit
| where count>=limit

The key components are match_type = WILDCARD(agent) in transforms.conf and *,7000. The former tells Splunk that it should treat * in a lookup file as a wildcard and thus allow partial matches. The latter is a default limit, applied when no other agent matched. That default lookup needs to be last in the lookup file.

View solution in original post

micahkemp
Champion

I think you'd be best served using a lookup table to define the limits, then filtering those where the count is greater than the limit with where. Example configuration below.

agent_limits.csv:

agent,limit
nfc*,700
mm_write,5000
breeze,1500
megafon_bitmap,1000
bankm_cashback,5000
*,7000

transforms.conf:

[agent_limits]
filename = agent_limits.csv
match_type = WILDCARD(agent)
max_matches = 1 

Run anywhere search demonstrating functionality:

| makeresults | eval agent="nfc1", count=650
| append [| makeresults | eval agent="nfc2", count=750]
| append [| makeresults | eval agent="breeze", count=5001]
| lookup agent_limits agent OUTPUT limit
| where count>=limit

The key components are match_type = WILDCARD(agent) in transforms.conf and *,7000. The former tells Splunk that it should treat * in a lookup file as a wildcard and thus allow partial matches. The latter is a default limit, applied when no other agent matched. That default lookup needs to be last in the lookup file.

exmuzzy
Explorer

Is there a way to create agent_limits.csv from the splunk web?

0 Karma

exmuzzy
Explorer

It works perfectly! Many thanks!

0 Karma

micahkemp
Champion

You can add lookups via Splunk Web, but you can't add the transforms I suggested via Web. Is this in Splunk Cloud, where there is no CLI access?

0 Karma

mayurr98
Super Champion

perhaps you can try something like this
its the same just took like terms together

  index=api | stats count as c by agent | search  (((agent!=breeze agent!=nfc* agent!=mm_write agent!=bankm_cashback agent!=megafon_bitmap) OR agent=nfc*) c>700) OR ((agent=mm_write OR agent=bankm_cashback) c>5000) OR (agent=breeze c>1500) OR (agent=megafon_bitmap c>1000) 

let me know if this helps!

0 Karma

exmuzzy
Explorer

2micahkemp: can you hint me how to calculate limits for agents based on their regular activity?
I mean, how to generate agent_limits.csv based on previous events automativcaly not manualy?

0 Karma

micahkemp
Champion

You can use | outputlookup to write search results to a lookup file, which would allow you to create it programmatically. My sample lookup file could have been produced by this search:

| makeresults | eval agent="nfc*", limit=700
| append [| makeresults | eval agent="mm_write", limit=5000]
| append [| makeresults | eval agent="breeze", limit=1500]
| append [| makeresults | eval agent="megafon_bitmap", limit=1000]
| append [| makeresults | eval agent="bankm_cashback", limit=5000]
| append [| makeresults | eval agent="*", limit=7000]
| fields - _time
| outputlookup agent_limits.csv

But the logic that determines what those limits should be based on your data, I don't know how to help you there.

0 Karma

exmuzzy
Explorer
0 Karma
Get Updates on the Splunk Community!

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

[Coming Soon] Splunk Observability Cloud - Enhanced navigation with a modern look and ...

We are excited to introduce our enhanced UI that brings together AppDynamics and Splunk Observability. This is ...