Alerting

Dynamic Server list?

mmcarty
New Member

Hello
this is the scenario, right now we receive an alert once our indexers capacity is above 85%
we do this by an alert

| rest /services/data/indexes search="totalEventCount!=0" | eval coldPath.maxDataSizeMB=if('coldPath.maxDataSizeMB' = 0, null(), 'coldPath.maxDataSizeMB') | eval homePath.maxDataSizeMB=if('homePath.maxDataSizeMB' = 0, null(), 'homePath.maxDataSizeMB') | eval roof=min((coalesce('homePath.maxDataSizeMB', 4294967295) +                  coalesce('coldPath.maxDataSizeMB', 4294967295)),                 maxTotalDataSizeMB) | eval span=tostring(currentDBSizeMB) + " / " +     tostring(roof) + " MB" | eval Percent=tostring(round(currentDBSizeMB * 100 / roof)) | where Percent > 85
|search [**inputlookup all_servers.csv** | search role=indexer | rename host AS splunk_server | fields splunk_server] | stats first(span) AS "Capacity vs Limit" by splunk_server title minTime maxTime Percent | rename splunk_server AS Indexer title AS Index minTime AS "Oldest Event" maxTime AS "Newest Event" | table Indexer Index "Capacity vs Limit" "Oldest Event" "Newest Event" Percent
| sort - Percent

as you can see we are using the inputlookup all_servers.csv
to define all the servers

We were ask, what about new servers are onboarded and they are not being updated on the list?
what if a new server is named incorrectly on the inputlookup.

the question is,
is there a way to tell splunk,
a file or anything to tell.
a new indexer is onboarded, pick it up and make it part of an alert if indexing capacity is beyond 85% trigger an alert?

Thank you!

Tags (2)
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Your Monitoring Console already has such a lookup, it's called dmc_assets and has a field search_group containing "dmc_group_indexer" for indexers.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

It's updated as part of your - hopefully existing - routine to use the monitoring console: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Configureindistributedmode

Note, the monitoring console already comes with alerts that are close to yours, albeit monitoring disk usage rather than index usage: http://docs.splunk.com/Documentation/Splunk/7.0.2/DMC/Platformalerts#Which_alerts_are_included.3F
You might be able to clone and modify that accordingly though, depending on what your actual use case is.

0 Karma

mmcarty
New Member

First of all, thank you very much for replying, I appreciate it, I would like to kindly ask what is the next step then? what i should do or modify with this lookup to take new indexers automatically?

Best regards.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Quantify Your Splunk Investment Impact: Introducing Savings Metrics to Value Insights

Building on the foundation established in our initial Value Insights releases, we are introducing the Savings ...

Event Series: Telemetry Pipeline Management

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...