Reporting

Is there a way I can be notified(via email) if splunkd stops when monitoring Splunkd services?

Mansi24
Path Finder

Hi,

My Splunkd service in production instance is automatically getting stopped. It has happened couple of times before as well. We don't get to know untill we or clients try logging in. Is there a way we can be notified if the services are getting stopped.

Please suggest.

0 Karma

gcusello
Esteemed Legend

Hi Mansi24,
if you're speaking of splunkd of a Universal Forwarder it's easy.
If instead you're speaking of splunkd of a Search Head, an Indexer or another Splunk server, it's possible only if you have a redundant architecture: in other words, you can detect that an Indexer is down only if you have another Indexer up otherwise it isn't possible.

Anyway, to detect down servers you have to create a lookup with all the servers to monitor (e.g. perimeter.csv) in which there a field called host; then you have to run a search like this and associate it to an alert to schedule e.g. every five minutes:

index=_internal earliest=-5m latest=now
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=lower(host) | fields count host ]
| stats sum(count) AS total BY host
| where total=0

You can use this search both for Universal Forwarders and/or Splunk servers, it depends by the perimeter you define.

Ciao.
Giuseppe

0 Karma

Mansi24
Path Finder

Thanks for the response gcusello!

Yes i am talking about my Search head here, forgot to mention that. Correct me if i am wrong, i need to create a lookup having all indexers and search head listed and try running this query. Would it work if my SH is down?

0 Karma

gcusello
Esteemed Legend

Hi Mansi24,
If you have only one Search Haed this alert cannot run because alerts run on SHs.
If you have more than one Search Head, you can schedule this alert on all of them so you'll have your alert.

Ciao.
Giuseppe

0 Karma

Mansi24
Path Finder

just an update i am using a shell script on my linux server where SH is hosted, which checks the splunk service every 5 minutes and starts it if not running.Also sends me email when not running.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...