forwarders restarting

[JarrettM]

Can anyone think of a reason that might cause all 32 of my Universal Forwarders to restart within a minute of 3:46 PM on Friday? The first mention of this in all splunkd logs is the essentially the same

01-19-2018 15:46:48.460 -0500 INFO DeployedServerclass - Serverclass=Airwatch is uninstalling app=E:\SplunkUniversalForwarder\etc\apps\IIS
01-19-2018 15:46:48.460 -0500 INFO DeployedApplication - Removing app=IIS at='E:\SplunkUniversalForwarder\etc\apps\IIS'
01-19-2018 15:46:48.460 -0500 INFO DeployedServerclass - Serverclass=Airwatch is uninstalling app=E:\SplunkUniversalForwarder\etc\apps\Perfmon
01-19-2018 15:46:48.460 -0500 INFO DeployedApplication - Removing app=Perfmon at='E:\SplunkUniversalForwarder\etc\apps\Perfmon'
01-19-2018 15:46:48.460 -0500 INFO DeployedServerclass - Serverclass=Airwatch is uninstalling app=E:\SplunkUniversalForwarder\etc\apps\WinEvt_Logs
01-19-2018 15:46:48.460 -0500 INFO DeployedApplication - Removing app=WinEvt_Logs at='E:\SplunkUniversalForwarder\etc\apps\WinEvt_Logs'
01-19-2018 15:46:48.491 -0500 WARN DC:DeploymentClient - Restarting Splunkd...

There is nothing in any of the Windows logs that show anything
unusual happening at this time.

[nickhills]

On the face of it, that looks like someone changed an entry in serverclass.conf at some point previously, and at 15:46 the deployment server restarted, pushing out the changes to your deployment clients.

Take a look at the logs on your DS, and see if you can work out if the deployment server was reloaded by hand, or restarted for some other reason

[JarrettM]

Thanks but that doesn't seem to be it. Server.conf isn't being deployed in any of the apps and the Deployment Server did not restart.

[nickhills]

Try searching for:
index=_internal sourctype=splunkd "DeploymentServer - Attempting to reload entire DS"
5+/- minutes around the time in question

[JarrettM]

Yes. In that minute all my server classes and apps have events similar to this one:

1/19/18
3:46:22.873 PM

01-19-2018 15:46:22.873 -0500 INFO DeploymentServer - Attempting to reload serverclass='Airwatch'; reason='(app=WinEvt_Logs) DeploymentServer::deinstallApplication'
host = HQTM-USPLNK-401 source = E:\Splunk\var\log\splunk\splunkd.log sourcetype = splunkd

But that still begs the question of WHY the Deployment Splunk server decided to reload and reinstall all the classes and apps.

[nickhills]

So it looks like your DS is on windows. Do you also use it as a search head, with the windows TA? At a guess I would say that a change was made in the ta config which triggered the DS to reload its config, and restart the clients.

[JarrettM]

Not using the Windows TA but somthing happened to the indexer at 3:38 PM on Friday. The index.conf file shows it was updated at 3:38:22 and the splunkd log shows these events:

1/19/18
3:38:22.243 PM

01-19-2018 15:38:22.243 -0500 INFO IndexProcessor - reloading index config: end
host = HQTM-USPLNK-401 source = E:\Splunk\var\log\splunk\splunkd.log sourcetype = splunkd
1/19/18
3:38:22.243 PM

01-19-2018 15:38:22.243 -0500 INFO IndexProcessor - Reloading index config: shutdown subordinate threads, now restarting
host = HQTM-USPLNK-401 source = E:\Splunk\var\log\splunk\splunkd.log sourcetype = splunkd
1/19/18
3:38:22.243 PM

01-19-2018 15:38:22.243 -0500 INFO IndexProcessor - reloading index config: start
host = HQTM-USPLNK-401 source = E:\Splunk\var\log\splunk\splunkd.log sourcetype = splunkd
1/19/18
3:38:22.233 PM

01-19-2018 15:38:22.233 -0500 INFO IndexerIf - reloading index config: request received

If any change was made I'm the only one who could have done it. We are just in the process of initial setup of the Splunk environment and I'm the only one with access. So it looks like I did something Friday afternoon but I have no idea what.

Thanks for your help!

[nickhills]

Hmm the timestamps are close enough to be more than coincidence.
You don't have any files named "crash" in your ./splunk/var/log/splunk directory?

[JarrettM]

No, no files named crash.