topic Re: Dynamic Server list? in Alerting

Dynamic Server list?

mmcarty — Sat, 24 Feb 2018 01:10:36 GMT

Hello
this is the scenario, right now we receive an alert once our indexers capacity is above 85%
we do this by an alert

| rest /services/data/indexes search="totalEventCount!=0" | eval coldPath.maxDataSizeMB=if('coldPath.maxDataSizeMB' = 0, null(), 'coldPath.maxDataSizeMB') | eval homePath.maxDataSizeMB=if('homePath.maxDataSizeMB' = 0, null(), 'homePath.maxDataSizeMB') | eval roof=min((coalesce('homePath.maxDataSizeMB', 4294967295) +                  coalesce('coldPath.maxDataSizeMB', 4294967295)),                 maxTotalDataSizeMB) | eval span=tostring(currentDBSizeMB) + " / " +     tostring(roof) + " MB" | eval Percent=tostring(round(currentDBSizeMB * 100 / roof)) | where Percent > 85
|search [**inputlookup all_servers.csv** | search role=indexer | rename host AS splunk_server | fields splunk_server] | stats first(span) AS "Capacity vs Limit" by splunk_server title minTime maxTime Percent | rename splunk_server AS Indexer title AS Index minTime AS "Oldest Event" maxTime AS "Newest Event" | table Indexer Index "Capacity vs Limit" "Oldest Event" "Newest Event" Percent
| sort - Percent

as you can see we are using the inputlookup all_servers.csv
to define all the servers

We were ask, what about new servers are onboarded and they are not being updated on the list?
what if a new server is named incorrectly on the inputlookup.

the question is,
is there a way to tell splunk,
a file or anything to tell.
a new indexer is onboarded, pick it up and make it part of an alert if indexing capacity is beyond 85% trigger an alert?

Thank you!

Re: Dynamic Server list?

martin_mueller — Sat, 24 Feb 2018 02:00:41 GMT

Your Monitoring Console already has such a lookup, it's called dmc_assets and has a field search_group containing "dmc_group_indexer" for indexers.

Re: Dynamic Server list?

mmcarty — Sat, 24 Feb 2018 15:07:19 GMT

First of all, thank you very much for replying, I appreciate it, I would like to kindly ask what is the next step then? what i should do or modify with this lookup to take new indexers automatically?

Best regards.

Re: Dynamic Server list?

martin_mueller — Sat, 24 Feb 2018 15:15:57 GMT

It's updated as part of your - hopefully existing - routine to use the monitoring console: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Configureindistributedmode

Note, the monitoring console already comes with alerts that are close to yours, albeit monitoring disk usage rather than index usage: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Platformalerts#Which_alerts_are_included.3F
You might be able to clone and modify that accordingly though, depending on what your actual use case is.