Splunk Search

How to get notified for indexer automatic detention

jpillai
Path Finder

Im not seeing any way Splunk will notify regarding automatic detention, which usually happens because of disk space issues

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

I think that the easiest way to get this is an activate MC's Alert: "DMC Alert - Abnormal State of Indexer Processor". Then it informs you e.g. with email that there is issue with indexer.

r. Ismo

0 Karma

jpillai
Path Finder

Thank you for the quick responses, I will certainly check on this and let you know. But it is a shame there is no better/obvious ways to do this in Splunk, given this is an important event.

0 Karma

sumanssah
Communicator

If looking for Splunk disk space issue, you can use your monitoring console for space details

Splunk >> Setting >> Monitoring Console >> Indexing >> Volume Detail Instance

or try below mentioned SPL

| rest /services/server/status/partitions-space | eval usage = capacity
- free | eval pct_usage = round(usage / capacity * 100, 2) | stats
first(fs_type) as fs_type first(usage) as usage first(capacity) as
capacity first(pct_usage) as pct_usage by splunk_server, mount_point

If you want to monitor Space issue in non-Splunk instances, try onboarding perfmon logs, onboarding details are on below mentioned Splunk answer

https://answers.splunk.com/answers/454999/how-to-develop-a-search-to-find-free-disk-space-us.html

index=perfmon sourcetype="Perfmon:Free Disk Space" counter="Free Megabytes" (instance!="HarddiskVolume*") (instance!=_Total) |dedup host
 | eval FreeSpace=(Value/1024)
 | eval GB=tostring(FreeSpace,"commas")
 | table host instance GB 
 | sort + host instance
 | rename instance as "Drive Letter" GB as "GigaBytes Free"
0 Karma

somesoni2
SplunkTrust
SplunkTrust

The cluster master dashboard shows status of indexers (via REST call). You can setup an alert to which can query the indexer status at an interval and notify you when the status changes to AutomaticDetention. See this for rest api endpoint details.

https://docs.splunk.com/Documentation/Splunk/8.0.2/RESTREF/RESTcluster#cluster.2Fmaster.2Fpeers

In search query you could do something like this (run on cluster master node)

| rest splunk_server=local /services/cluster/master/peers | table title label status | where status="AutomaticDetention"
0 Karma

jpillai
Path Finder

@somesoni2 This is exactly what I was looking for, once I found there is no user friendly way of doing it. I have setup an alert with this on the master. Thank you 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...