Below is the scenario:
We would like to implement an alert which would be triggered if a particular index — or indexes — started flooding suddenly (Consider an environment which has more than 3k indexes)
Can anyone please help us on this?
Regards,
Abilash S
Why not just measure for an unusually heavy source? ForwarderLevel - Splunk Heavy logging sources in https://github.com/gjanders/SplunkAdmins/blob/master/default/savedsearches.conf
Or the equivalent alert in Alerts For Splunk Admins
You could also use tstats to baseline over time the number of events and look for excessive numbers...
| tstats count where index=* groupby index
But that would require a lookup file or kvstore and some logic of course
Hi @kishor_pinjarkar ,
I've run the given query and got "Search Factory: Unknown search command dmc".
Could you please check this error once?
Thank you..!
Are you looking for this:
Alert:
| rest splunk_server=### /services/data/indexes datatype=all
| join title type=outer
[| rest splunk_server=### /services/data/indexes-extended datatype=all
| fields title, total_bucket_count]
| `dmc_exclude_indexes`
| fields title maxTotalDataSizeMB currentDBSizeMB
| eval currentDBSizeGB = if(isnotnull(currentDBSizeMB), round(currentDBSizeMB / 1024, 2), 0)
| eval maxTotalDataSizeGB = if((maxTotalDataSizeMB == 0) OR isnull(maxTotalDataSizeMB), "unlimited", round(maxTotalDataSizeMB / 1024, 2))
| eval percused = round((currentDBSizeMB / maxTotalDataSizeMB) *100,2)
| fields - maxTotalDataSizeMB currentDBSizeMB
Result:
title currentDBSizeGB maxTotalDataSizeGB percused
_audit 15.89 488.28 3.25
_internal 487.35 488.28 99.81
_introspection 3.53 488.28 0.72
If percused > threshold, then trigger alert...
Note:
Replace ### with your hostname.
When you put a search macro in a search string, place a back tick character (`) before and after the macro name. On most English-language keyboards, this character is located on the same key as the tilde (~).
dmc_exclude_indexes
Hi @kishor_pinjarkar,
It worked out..! but i have been encountering the below error now.
REST Processor: Failed to fetch REST endpoint uri=http://127.0.0.1:8089/services/data/indexes?count=0&datatype=all from server http://127.0.0.1:8089. Check that the URI path provided exists in the REST API
Can you please have a look on this.?
Thank you.!
can you try
splunk_server=local
in both the places...
No luck. getting the same error
What is average indexing rate in your case?
We can make a threshold for this one and if that threshold get touched we can create a alert.
Please provide more details...
I can help you with that.
Hi @kishor_pinjarkar ,
Average indexing rate would be 150 - 200 GB.
Whenever the index rate is high , we need to find out the index which is flooding and notify the end user.
Thank you,
Abilash S
Are you talking about index size, per day, which is 150-200 GB
or
indexing rate which will be KB/s?
Yes, Index usage is 150 - 200 GB.