Hi, I am deploying apps from deployment server.
Server classes having restartSplunkd=1 gets to stop when I deploy new apps but it doesn't start app.
How does the deployment server send a request to remote client?
is there any way to see what exactly going wrong here?
After checking splunkd.log
for errors, also check for any typo's in your configuration files. Splunk doesn't start if it encounters any typo's or mistakes in critical configuration settings. You can use btool to troubleshoot your configuration files. HTH!
If I restart the splunkd manually on the client side, I can start it up so I don't think it's typo or any configuration error.
This is the client that just got new app deployed. it received signal from deployment server and shutdown itself successfully.
it just doesn't come back up again.
VVV
tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log 07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_LoadLDAPUsers"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_MetricsManager"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Pipeline"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Queue"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_CallbackRunner"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_HttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_DmcProxyHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Duo2FAHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - Shutdown complete in
1561.8 milliseconds
07-25-2018 21:03:58.322 +0000 INFO loader - All pipelines finished.
Try looking for information in splunkd_stderr.log
and splunkd_stdout.log
. These log files contains some information about startup process.
PS: File path is same, /opt/splunkforwarder/var/log/splunk/
You can check your _internal logs to see if there are any issues during deployment. Does Splunk get restarted on the forwarder when its deployed? Are there any errors during restart?
Does Splunk get restarted on the forwarder when its deployed?
No. I mentioned " it doesn't start app" but it's actually not starting up the forwarder process itself.
Are there any errors during restart?
I didn't see anything unusual in _internal index. Is there any particular component or log file name I should look into for deployment server behavior?