Alerting

How to create alert with customized limits for many agents?

exmuzzy
Explorer

I need to be alerted that the agent is exceeding its specified limit, and if the agent limit is not set, then the base limit triggered

To atchive that I have created such search string for alert

index=api | stats count as c by agent | search  (agent=nfc* c>700) OR (agent=mm_write c>5000) OR (agent=breeze c>1500) OR (agent=megafon_bitmap c > 1000) OR (agent=bankm_cashback с>5000)  OR (agent!=breeze agent!=nfc* agent!=mm_write agent!=bankm_cashback agent!=megafon_bitmap c>700)  

But it looks too cumbersome

Is there better way to do this?
I would like to maintain some kind of table of agents limits.

0 Karma
1 Solution

micahkemp
Champion

I think you'd be best served using a lookup table to define the limits, then filtering those where the count is greater than the limit with where. Example configuration below.

agent_limits.csv:

agent,limit
nfc*,700
mm_write,5000
breeze,1500
megafon_bitmap,1000
bankm_cashback,5000
*,7000

transforms.conf:

[agent_limits]
filename = agent_limits.csv
match_type = WILDCARD(agent)
max_matches = 1 

Run anywhere search demonstrating functionality:

| makeresults | eval agent="nfc1", count=650
| append [| makeresults | eval agent="nfc2", count=750]
| append [| makeresults | eval agent="breeze", count=5001]
| lookup agent_limits agent OUTPUT limit
| where count>=limit

The key components are match_type = WILDCARD(agent) in transforms.conf and *,7000. The former tells Splunk that it should treat * in a lookup file as a wildcard and thus allow partial matches. The latter is a default limit, applied when no other agent matched. That default lookup needs to be last in the lookup file.

View solution in original post

micahkemp
Champion

I think you'd be best served using a lookup table to define the limits, then filtering those where the count is greater than the limit with where. Example configuration below.

agent_limits.csv:

agent,limit
nfc*,700
mm_write,5000
breeze,1500
megafon_bitmap,1000
bankm_cashback,5000
*,7000

transforms.conf:

[agent_limits]
filename = agent_limits.csv
match_type = WILDCARD(agent)
max_matches = 1 

Run anywhere search demonstrating functionality:

| makeresults | eval agent="nfc1", count=650
| append [| makeresults | eval agent="nfc2", count=750]
| append [| makeresults | eval agent="breeze", count=5001]
| lookup agent_limits agent OUTPUT limit
| where count>=limit

The key components are match_type = WILDCARD(agent) in transforms.conf and *,7000. The former tells Splunk that it should treat * in a lookup file as a wildcard and thus allow partial matches. The latter is a default limit, applied when no other agent matched. That default lookup needs to be last in the lookup file.

exmuzzy
Explorer

Is there a way to create agent_limits.csv from the splunk web?

0 Karma

exmuzzy
Explorer

It works perfectly! Many thanks!

0 Karma

micahkemp
Champion

You can add lookups via Splunk Web, but you can't add the transforms I suggested via Web. Is this in Splunk Cloud, where there is no CLI access?

0 Karma

mayurr98
Super Champion

perhaps you can try something like this
its the same just took like terms together

  index=api | stats count as c by agent | search  (((agent!=breeze agent!=nfc* agent!=mm_write agent!=bankm_cashback agent!=megafon_bitmap) OR agent=nfc*) c>700) OR ((agent=mm_write OR agent=bankm_cashback) c>5000) OR (agent=breeze c>1500) OR (agent=megafon_bitmap c>1000) 

let me know if this helps!

0 Karma

exmuzzy
Explorer

2micahkemp: can you hint me how to calculate limits for agents based on their regular activity?
I mean, how to generate agent_limits.csv based on previous events automativcaly not manualy?

0 Karma

micahkemp
Champion

You can use | outputlookup to write search results to a lookup file, which would allow you to create it programmatically. My sample lookup file could have been produced by this search:

| makeresults | eval agent="nfc*", limit=700
| append [| makeresults | eval agent="mm_write", limit=5000]
| append [| makeresults | eval agent="breeze", limit=1500]
| append [| makeresults | eval agent="megafon_bitmap", limit=1000]
| append [| makeresults | eval agent="bankm_cashback", limit=5000]
| append [| makeresults | eval agent="*", limit=7000]
| fields - _time
| outputlookup agent_limits.csv

But the logic that determines what those limits should be based on your data, I don't know how to help you there.

0 Karma

exmuzzy
Explorer
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...