Alerting
Highlighted

Can you help me build an alert which returns when there's a sudden increase in the index rate?

Explorer

Below is the scenario:

We would like to implement an alert which would be triggered if a particular index — or indexes — started flooding suddenly (Consider an environment which has more than 3k indexes)

Can anyone please help us on this?

Regards,
Abilash S

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Path Finder

What is average indexing rate in your case?
We can make a threshold for this one and if that threshold get touched we can create a alert.

Please provide more details...

I can help you with that.

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Explorer

Hi @kishor_pinjarkar ,

Average indexing rate would be 150 - 200 GB.
Whenever the index rate is high , we need to find out the index which is flooding and notify the end user.

Thank you,
Abilash S

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Path Finder

Are you talking about index size, per day, which is 150-200 GB
or
indexing rate which will be KB/s?

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Explorer

Yes, Index usage is 150 - 200 GB.

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Path Finder

Are you looking for this:

Alert:

| rest splunk_server=###  /services/data/indexes datatype=all 
| join title type=outer 
    [| rest splunk_server=### /services/data/indexes-extended datatype=all 
    | fields title, total_bucket_count] 
| `dmc_exclude_indexes` 
| fields title maxTotalDataSizeMB currentDBSizeMB 
| eval currentDBSizeGB = if(isnotnull(currentDBSizeMB), round(currentDBSizeMB / 1024, 2), 0) 
| eval maxTotalDataSizeGB = if((maxTotalDataSizeMB == 0) OR isnull(maxTotalDataSizeMB), "unlimited", round(maxTotalDataSizeMB / 1024, 2)) 
| eval percused = round((currentDBSizeMB / maxTotalDataSizeMB) *100,2) 
| fields - maxTotalDataSizeMB currentDBSizeMB

Result:

title   currentDBSizeGB maxTotalDataSizeGB  percused
_audit  15.89             488.28                            3.25
_internal   487.35    488.28                            99.81
_introspection  3.53   488.28                           0.72

If percused > threshold, then trigger alert...

Note:
Replace ### with your hostname.

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Path Finder

When you put a search macro in a search string, place a back tick character (`) before and after the macro name. On most English-language keyboards, this character is located on the same key as the tilde (~).

dmc_exclude_indexes

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Explorer

Hi @kishor_pinjarkar,

It worked out..! but i have been encountering the below error now.

REST Processor: Failed to fetch REST endpoint uri=http://127.0.0.1:8089/services/data/indexes?count=0&datatype=all from server http://127.0.0.1:8089. Check that the URI path provided exists in the REST API

Can you please have a look on this.?

Thank you.!

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Path Finder

can you try

splunk_server=local

in both the places...

0 Karma
Highlighted

Re: Can you help me build an alert which returns when there's a sudden increase in the index rate?

Explorer

No luck. getting the same error

0 Karma