Alerting

Dynamic Server list?

mmcarty
New Member

Hello
this is the scenario, right now we receive an alert once our indexers capacity is above 85%
we do this by an alert

| rest /services/data/indexes search="totalEventCount!=0" | eval coldPath.maxDataSizeMB=if('coldPath.maxDataSizeMB' = 0, null(), 'coldPath.maxDataSizeMB') | eval homePath.maxDataSizeMB=if('homePath.maxDataSizeMB' = 0, null(), 'homePath.maxDataSizeMB') | eval roof=min((coalesce('homePath.maxDataSizeMB', 4294967295) +                  coalesce('coldPath.maxDataSizeMB', 4294967295)),                 maxTotalDataSizeMB) | eval span=tostring(currentDBSizeMB) + " / " +     tostring(roof) + " MB" | eval Percent=tostring(round(currentDBSizeMB * 100 / roof)) | where Percent > 85
|search [**inputlookup all_servers.csv** | search role=indexer | rename host AS splunk_server | fields splunk_server] | stats first(span) AS "Capacity vs Limit" by splunk_server title minTime maxTime Percent | rename splunk_server AS Indexer title AS Index minTime AS "Oldest Event" maxTime AS "Newest Event" | table Indexer Index "Capacity vs Limit" "Oldest Event" "Newest Event" Percent
| sort - Percent

as you can see we are using the inputlookup all_servers.csv
to define all the servers

We were ask, what about new servers are onboarded and they are not being updated on the list?
what if a new server is named incorrectly on the inputlookup.

the question is,
is there a way to tell splunk,
a file or anything to tell.
a new indexer is onboarded, pick it up and make it part of an alert if indexing capacity is beyond 85% trigger an alert?

Thank you!

Tags (2)
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Your Monitoring Console already has such a lookup, it's called dmc_assets and has a field search_group containing "dmc_group_indexer" for indexers.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

It's updated as part of your - hopefully existing - routine to use the monitoring console: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Configureindistributedmode

Note, the monitoring console already comes with alerts that are close to yours, albeit monitoring disk usage rather than index usage: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Platformalerts#Which_alerts_are_included.3F
You might be able to clone and modify that accordingly though, depending on what your actual use case is.

0 Karma

mmcarty
New Member

First of all, thank you very much for replying, I appreciate it, I would like to kindly ask what is the next step then? what i should do or modify with this lookup to take new indexers automatically?

Best regards.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...