topic Re: forwarders restarting in Deployment Architecture

forwarders restarting

JarrettM — Tue, 29 Sep 2020 17:45:59 GMT

Can anyone think of a reason that might cause all 32 of my Universal Forwarders to restart within a minute of 3:46 PM on Friday? The first mention of this in all splunkd logs is the essentially the same

01-19-2018 15:46:48.460 -0500 INFO DeployedServerclass - Serverclass=Airwatch is uninstalling app=E:\SplunkUniversalForwarder\etc\apps\IIS
01-19-2018 15:46:48.460 -0500 INFO DeployedApplication - Removing app=IIS at='E:\SplunkUniversalForwarder\etc\apps\IIS'
01-19-2018 15:46:48.460 -0500 INFO DeployedServerclass - Serverclass=Airwatch is uninstalling app=E:\SplunkUniversalForwarder\etc\apps\Perfmon
01-19-2018 15:46:48.460 -0500 INFO DeployedApplication - Removing app=Perfmon at='E:\SplunkUniversalForwarder\etc\apps\Perfmon'
01-19-2018 15:46:48.460 -0500 INFO DeployedServerclass - Serverclass=Airwatch is uninstalling app=E:\SplunkUniversalForwarder\etc\apps\WinEvt_Logs
01-19-2018 15:46:48.460 -0500 INFO DeployedApplication - Removing app=WinEvt_Logs at='E:\SplunkUniversalForwarder\etc\apps\WinEvt_Logs'
01-19-2018 15:46:48.491 -0500 WARN DC:DeploymentClient - Restarting Splunkd...

There is nothing in any of the Windows logs that show anything
unusual happening at this time.

Re: forwarders restarting

nickhills — Mon, 22 Jan 2018 15:24:26 GMT

On the face of it, that looks like someone changed an entry in serverclass.conf at some point previously, and at 15:46 the deployment server restarted, pushing out the changes to your deployment clients.

Take a look at the logs on your DS, and see if you can work out if the deployment server was reloaded by hand, or restarted for some other reason

Re: forwarders restarting

JarrettM — Mon, 22 Jan 2018 16:19:33 GMT

Thanks but that doesn't seem to be it. Server.conf isn't being deployed in any of the apps and the Deployment Server did not restart.

Re: forwarders restarting

nickhills — Mon, 22 Jan 2018 16:36:55 GMT

Try searching for:
index=_internal sourctype=splunkd "DeploymentServer - Attempting to reload entire DS"
5+/- minutes around the time in question

Re: forwarders restarting

JarrettM — Mon, 22 Jan 2018 17:05:20 GMT

Yes. In that minute all my server classes and apps have events similar to this one:

1/19/18
3:46:22.873 PM

01-19-2018 15:46:22.873 -0500 INFO DeploymentServer - Attempting to reload serverclass='Airwatch'; reason='(app=WinEvt_Logs) DeploymentServer::deinstallApplication'
host = HQTM-USPLNK-401 source = E:\Splunk\var\log\splunk\splunkd.log sourcetype = splunkd

But that still begs the question of WHY the Deployment Splunk server decided to reload and reinstall all the classes and apps.

Re: forwarders restarting

nickhills — Mon, 22 Jan 2018 17:20:20 GMT

So it looks like your DS is on windows. Do you also use it as a search head, with the windows TA? At a guess I would say that a change was made in the ta config which triggered the DS to reload its config, and restart the clients.

Re: forwarders restarting

JarrettM — Mon, 22 Jan 2018 19:14:12 GMT

Not using the Windows TA but somthing happened to the indexer at 3:38 PM on Friday. The index.conf file shows it was updated at 3:38:22 and the splunkd log shows these events:

1/19/18
3:38:22.243 PM

01-19-2018 15:38:22.243 -0500 INFO IndexProcessor - reloading index config: end
host = HQTM-USPLNK-401 source = E:\Splunk\var\log\splunk\splunkd.log sourcetype = splunkd
1/19/18
3:38:22.243 PM

01-19-2018 15:38:22.243 -0500 INFO IndexProcessor - Reloading index config: shutdown subordinate threads, now restarting
host = HQTM-USPLNK-401 source = E:\Splunk\var\log\splunk\splunkd.log sourcetype = splunkd
1/19/18
3:38:22.243 PM

01-19-2018 15:38:22.243 -0500 INFO IndexProcessor - reloading index config: start
host = HQTM-USPLNK-401 source = E:\Splunk\var\log\splunk\splunkd.log sourcetype = splunkd
1/19/18
3:38:22.233 PM

01-19-2018 15:38:22.233 -0500 INFO IndexerIf - reloading index config: request received

If any change was made I'm the only one who could have done it. We are just in the process of initial setup of the Splunk environment and I'm the only one with access. So it looks like I did something Friday afternoon but I have no idea what.

Thanks for your help!

Re: forwarders restarting

nickhills — Mon, 22 Jan 2018 19:40:20 GMT

Hmm the timestamps are close enough to be more than coincidence.
You don't have any files named "crash" in your ./splunk/var/log/splunk directory?

Re: forwarders restarting

JarrettM — Tue, 23 Jan 2018 13:00:08 GMT

No, no files named crash.