Can you help me build an alert which returns when there's a sudden increase in the index rate?

[AbilashSe]

Below is the scenario:

We would like to implement an alert which would be triggered if a particular index — or indexes — started flooding suddenly (Consider an environment which has more than 3k indexes)

Can anyone please help us on this?

Regards,
Abilash S

[gjanders]

Why not just measure for an unusually heavy source? ForwarderLevel - Splunk Heavy logging sources in https://github.com/gjanders/SplunkAdmins/blob/master/default/savedsearches.conf

Or the equivalent alert in Alerts For Splunk Admins

You could also use tstats to baseline over time the number of events and look for excessive numbers...
| tstats count where index=* groupby index
But that would require a lookup file or kvstore and some logic of course

[AbilashSe]

Hi @kishor_pinjarkar ,

I've run the given query and got "Search Factory: Unknown search command dmc".
Could you please check this error once?

Thank you..!

[kishor_pinjark2]

Are you looking for this:

Alert:

| rest splunk_server=###  /services/data/indexes datatype=all 
| join title type=outer 
    [| rest splunk_server=### /services/data/indexes-extended datatype=all 
    | fields title, total_bucket_count] 
| `dmc_exclude_indexes` 
| fields title maxTotalDataSizeMB currentDBSizeMB 
| eval currentDBSizeGB = if(isnotnull(currentDBSizeMB), round(currentDBSizeMB / 1024, 2), 0) 
| eval maxTotalDataSizeGB = if((maxTotalDataSizeMB == 0) OR isnull(maxTotalDataSizeMB), "unlimited", round(maxTotalDataSizeMB / 1024, 2)) 
| eval percused = round((currentDBSizeMB / maxTotalDataSizeMB) *100,2) 
| fields - maxTotalDataSizeMB currentDBSizeMB

Result:

title   currentDBSizeGB maxTotalDataSizeGB  percused
_audit  15.89             488.28                            3.25
_internal   487.35    488.28                            99.81
_introspection  3.53   488.28                           0.72

If percused > threshold, then trigger alert...

Note:
Replace ### with your hostname.

[kishor_pinjark2]

When you put a search macro in a search string, place a back tick character (`) before and after the macro name. On most English-language keyboards, this character is located on the same key as the tilde (~).

dmc_exclude_indexes

[AbilashSe]

Hi @kishor_pinjarkar,

It worked out..! but i have been encountering the below error now.

REST Processor: Failed to fetch REST endpoint uri=http://127.0.0.1:8089/services/data/indexes?count=0&datatype=all from server http://127.0.0.1:8089. Check that the URI path provided exists in the REST API

Can you please have a look on this.?

Thank you.!

[kishor_pinjark2]

can you try

splunk_server=local

in both the places...

[AbilashSe]

No luck. getting the same error

[kishor_pinjark2]

What is average indexing rate in your case?
We can make a threshold for this one and if that threshold get touched we can create a alert.

Please provide more details...

I can help you with that.

[AbilashSe]

Hi @kishor_pinjarkar ,

Average indexing rate would be 150 - 200 GB.
Whenever the index rate is high , we need to find out the index which is flooding and notify the end user.

Thank you,
Abilash S

[kishor_pinjark2]

Are you talking about index size, per day, which is 150-200 GB
or
indexing rate which will be KB/s?

[AbilashSe]

Yes, Index usage is 150 - 200 GB.