Alerting

Dynamic Server list?

mmcarty
New Member

Hello
this is the scenario, right now we receive an alert once our indexers capacity is above 85%
we do this by an alert

| rest /services/data/indexes search="totalEventCount!=0" | eval coldPath.maxDataSizeMB=if('coldPath.maxDataSizeMB' = 0, null(), 'coldPath.maxDataSizeMB') | eval homePath.maxDataSizeMB=if('homePath.maxDataSizeMB' = 0, null(), 'homePath.maxDataSizeMB') | eval roof=min((coalesce('homePath.maxDataSizeMB', 4294967295) +                  coalesce('coldPath.maxDataSizeMB', 4294967295)),                 maxTotalDataSizeMB) | eval span=tostring(currentDBSizeMB) + " / " +     tostring(roof) + " MB" | eval Percent=tostring(round(currentDBSizeMB * 100 / roof)) | where Percent > 85
|search [**inputlookup all_servers.csv** | search role=indexer | rename host AS splunk_server | fields splunk_server] | stats first(span) AS "Capacity vs Limit" by splunk_server title minTime maxTime Percent | rename splunk_server AS Indexer title AS Index minTime AS "Oldest Event" maxTime AS "Newest Event" | table Indexer Index "Capacity vs Limit" "Oldest Event" "Newest Event" Percent
| sort - Percent

as you can see we are using the inputlookup all_servers.csv
to define all the servers

We were ask, what about new servers are onboarded and they are not being updated on the list?
what if a new server is named incorrectly on the inputlookup.

the question is,
is there a way to tell splunk,
a file or anything to tell.
a new indexer is onboarded, pick it up and make it part of an alert if indexing capacity is beyond 85% trigger an alert?

Thank you!

Tags (2)
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Your Monitoring Console already has such a lookup, it's called dmc_assets and has a field search_group containing "dmc_group_indexer" for indexers.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

It's updated as part of your - hopefully existing - routine to use the monitoring console: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Configureindistributedmode

Note, the monitoring console already comes with alerts that are close to yours, albeit monitoring disk usage rather than index usage: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Platformalerts#Which_alerts_are_included.3F
You might be able to clone and modify that accordingly though, depending on what your actual use case is.

0 Karma

mmcarty
New Member

First of all, thank you very much for replying, I appreciate it, I would like to kindly ask what is the next step then? what i should do or modify with this lookup to take new indexers automatically?

Best regards.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...