topic Re: detecting splunkd shutdown for forwarders? in Getting Data In

How to detect splunkd shutdown for forwarders?

remy06 — Tue, 25 Apr 2023 01:01:45 GMT

I may have missed out somewhere but I'm wondering if anyone has a way to detect if splunkd is being shutdown by an admin on forwarders.

The concern is that if I have a forwarders installed on a Windows/Linux server,an admin with access to the servers may still be able to stop splunk services,tamper with things and then restart the services(deployment monitor only checks if there are events missing within certain time..)

So is there anyway we can detect this?Thanks in advanced

Re: detecting splunkd shutdown for forwarders?

MuS — Mon, 22 Aug 2011 05:54:24 GMT

Hi remy06

just tested that and if you forward your forwarders _internal (outputs.conf - forwardedindex) to the indexer, you can see a message like this:

07:25:46.174 AM 08-22-2011 05:25:46.174 +0200 INFO ShutdownHandler - shutting down level ...

if you search index=_internal ShutDownHandler on your indexer. this way you would at least know someone stopped the forwarder.

regards,
MuS

Re: detecting splunkd shutdown for forwarders?

remy06 — Tue, 23 Aug 2011 08:57:23 GMT

Thanks.Also found out that it logs to windows system event log as well.Will be able to pull from there either way.

Re: detecting splunkd shutdown for forwarders?

remy06 — Thu, 08 Dec 2011 09:22:01 GMT

I've been using your suggestions and it works fine,so far if splunk was shut down using .../splunk start/stop on linux. However on linux the privileged user can also execute the "kill" command to stop the service.Wondering what other workarounds you guys have?

I've been trying to add a rule in auditd to do the work but it doesn't seem to work yet

Re: detecting splunkd shutdown for forwarders?

MuS — Thu, 08 Dec 2011 09:27:51 GMT

Hi remy06 in this case build a watchdog script on your linux box, which checks if the process 'splunkd' is running and if not restart 'splunkd' and your set 😉

find an example here: http://blog.eracc.com/2010/05/08/linux-monitor-a-service-with-a-watchdog-script/

cheers

Re: detecting splunkd shutdown for forwarders?

remy06 — Fri, 30 Dec 2011 07:33:54 GMT

Some updates,linux aside,I just did an upgrade for one of our old splunk forwarder(to universal forwarder 4.2.4) on windows server.I've noticed the following:

1) By searching the _internal index method,if the forwarder is shutdown,the indexer will not receive the shutdown event until the forwarder has restarted.(This way we probably will not be alerted when its down?)

2)By monitoring the windows system event log for forwarder shutdown events,when the service is shut down an event will be logged to windows event but the forwarder will not send this event to indexer.Even if the forwarder service has been restarted,the duration when the forwarder is down will not be captured at indexer.(I'm not sure why but seems that the older version of splunk able to do so)

Re: detecting splunkd shutdown for forwarders?

MuS — Tue, 03 Jan 2012 14:11:10 GMT

Hi remy06

maybe look at this problem more from the sysadmin scope rather than splunk admin scope. read this post at the very end http://www.indigorose.com/forums/archive/index.php/t-30167.html on how to make a service unstoppable.

don't blame me if you break something 🙂

Re: detecting splunkd shutdown for forwarders?

chimbudp — Tue, 18 Jun 2013 12:03:45 GMT

We can write a powershell script to monitor Windows- splunkd service.
This powershell script will monitor the service , and using SMTP server , we can trigger alert to mail ids , who support Splunk.

then immediate actions can be taken to start Splunk services back

Re: detecting splunkd shutdown for forwarders?

chimbudp — Tue, 18 Jun 2013 12:04:17 GMT

I have implemented this. If you need the script , i can help you,

Re: detecting splunkd shutdown for forwarders?

Eminemvin — Thu, 08 Sep 2016 03:51:48 GMT

i'm also trying to capture a stop command BEFORE splunk actually stops. the workaround I'm currently doing is editing the splunk script in init.d. i write something to file that the forwarder monitors before before the line that stops splunk. it somehow works but i'm still looking for a better way of implementing this without having to modify the default splunk files.

Re: detecting splunkd shutdown for forwarders?

tred23 — Thu, 19 Jan 2017 10:06:41 GMT

Here is the query I use
index=_internal component=ShutdownHandler sourcetype=splunkd message="Shutting down splunkd"

Re: detecting splunkd shutdown for forwarders?

sandyIscream — Sat, 03 Jun 2017 11:16:50 GMT

Can you paste the script ?

Re: detecting splunkd shutdown for forwarders?

youngsuh — Tue, 25 Apr 2023 00:54:01 GMT

index = _internal component=Shutdown | stats earliest(_time) as etime latest(_time) as ltime by host | convert timeformat="%Y/%m/%d %T" ctime(etime) ctime(ltime)

The SPL above work on version 9.x