Hello
this is the scenario, right now we receive an alert once our indexers capacity is above 85%
we do this by an alert
| rest /services/data/indexes search="totalEventCount!=0" | eval coldPath.maxDataSizeMB=if('coldPath.maxDataSizeMB' = 0, null(), 'coldPath.maxDataSizeMB') | eval homePath.maxDataSizeMB=if('homePath.maxDataSizeMB' = 0, null(), 'homePath.maxDataSizeMB') | eval roof=min((coalesce('homePath.maxDataSizeMB', 4294967295) + coalesce('coldPath.maxDataSizeMB', 4294967295)), maxTotalDataSizeMB) | eval span=tostring(currentDBSizeMB) + " / " + tostring(roof) + " MB" | eval Percent=tostring(round(currentDBSizeMB * 100 / roof)) | where Percent > 85
|search [**inputlookup all_servers.csv** | search role=indexer | rename host AS splunk_server | fields splunk_server] | stats first(span) AS "Capacity vs Limit" by splunk_server title minTime maxTime Percent | rename splunk_server AS Indexer title AS Index minTime AS "Oldest Event" maxTime AS "Newest Event" | table Indexer Index "Capacity vs Limit" "Oldest Event" "Newest Event" Percent
| sort - Percent
as you can see we are using the inputlookup all_servers.csv
to define all the servers
We were ask, what about new servers are onboarded and they are not being updated on the list?
what if a new server is named incorrectly on the inputlookup.
the question is,
is there a way to tell splunk,
a file or anything to tell.
a new indexer is onboarded, pick it up and make it part of an alert if indexing capacity is beyond 85% trigger an alert?
Thank you!
Your Monitoring Console already has such a lookup, it's called dmc_assets
and has a field search_group
containing "dmc_group_indexer"
for indexers.
It's updated as part of your - hopefully existing - routine to use the monitoring console: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Configureindistributedmode
Note, the monitoring console already comes with alerts that are close to yours, albeit monitoring disk usage rather than index usage: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Platformalerts#Which_alerts_are_included.3F
You might be able to clone and modify that accordingly though, depending on what your actual use case is.
First of all, thank you very much for replying, I appreciate it, I would like to kindly ask what is the next step then? what i should do or modify with this lookup to take new indexers automatically?
Best regards.