How to create alert with customized limits for many agents?

[exmuzzy]

I need to be alerted that the agent is exceeding its specified limit, and if the agent limit is not set, then the base limit triggered

To atchive that I have created such search string for alert

index=api | stats count as c by agent | search  (agent=nfc* c>700) OR (agent=mm_write c>5000) OR (agent=breeze c>1500) OR (agent=megafon_bitmap c > 1000) OR (agent=bankm_cashback с>5000)  OR (agent!=breeze agent!=nfc* agent!=mm_write agent!=bankm_cashback agent!=megafon_bitmap c>700)  

But it looks too cumbersome

Is there better way to do this?
I would like to maintain some kind of table of agents limits.

[micahkemp · Accepted solution]

I think you'd be best served using a lookup table to define the limits, then filtering those where the count is greater than the limit with where. Example configuration below.

agent_limits.csv:

agent,limit
nfc*,700
mm_write,5000
breeze,1500
megafon_bitmap,1000
bankm_cashback,5000
*,7000

transforms.conf:

[agent_limits]
filename = agent_limits.csv
match_type = WILDCARD(agent)
max_matches = 1 

Run anywhere search demonstrating functionality:

| makeresults | eval agent="nfc1", count=650
| append [| makeresults | eval agent="nfc2", count=750]
| append [| makeresults | eval agent="breeze", count=5001]
| lookup agent_limits agent OUTPUT limit
| where count>=limit

The key components are match_type = WILDCARD(agent) in transforms.conf and *,7000. The former tells Splunk that it should treat * in a lookup file as a wildcard and thus allow partial matches. The latter is a default limit, applied when no other agent matched. That default lookup needs to be last in the lookup file.

[exmuzzy]

Is there a way to create agent_limits.csv from the splunk web?

[exmuzzy]

It works perfectly! Many thanks!

[micahkemp]

You can add lookups via Splunk Web, but you can't add the transforms I suggested via Web. Is this in Splunk Cloud, where there is no CLI access?

[mayurr98]

perhaps you can try something like this
its the same just took like terms together

  index=api | stats count as c by agent | search  (((agent!=breeze agent!=nfc* agent!=mm_write agent!=bankm_cashback agent!=megafon_bitmap) OR agent=nfc*) c>700) OR ((agent=mm_write OR agent=bankm_cashback) c>5000) OR (agent=breeze c>1500) OR (agent=megafon_bitmap c>1000) 

let me know if this helps!

[exmuzzy]

2micahkemp: can you hint me how to calculate limits for agents based on their regular activity?
I mean, how to generate agent_limits.csv based on previous events automativcaly not manualy?

[micahkemp]

You can use | outputlookup to write search results to a lookup file, which would allow you to create it programmatically. My sample lookup file could have been produced by this search:

| makeresults | eval agent="nfc*", limit=700
| append [| makeresults | eval agent="mm_write", limit=5000]
| append [| makeresults | eval agent="breeze", limit=1500]
| append [| makeresults | eval agent="megafon_bitmap", limit=1000]
| append [| makeresults | eval agent="bankm_cashback", limit=5000]
| append [| makeresults | eval agent="*", limit=7000]
| fields - _time
| outputlookup agent_limits.csv

But the logic that determines what those limits should be based on your data, I don't know how to help you there.

[exmuzzy]

Thanks a lot!
I've created another question on this topic
https://answers.splunk.com/answers/611366/how-to-calculate-limits-for-agents-based-on-their.html