restartSplunkd doesn't restart Splunkd

[akadoya]

Hi, I am deploying apps from deployment server.
Server classes having restartSplunkd=1 gets to stop when I deploy new apps but it doesn't start app.

How does the deployment server send a request to remote client?
is there any way to see what exactly going wrong here?

[sudosplunk]

After checking splunkd.log for errors, also check for any typo's in your configuration files. Splunk doesn't start if it encounters any typo's or mistakes in critical configuration settings. You can use btool to troubleshoot your configuration files. HTH!

[akadoya]

If I restart the splunkd manually on the client side, I can start it up so I don't think it's typo or any configuration error.

This is the client that just got new app deployed. it received signal from deployment server and shutdown itself successfully.
it just doesn't come back up again.
VVV

tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log
07-25-2018 21:03:57.382 +0000 INFO  ShutdownHandler - shutting down level

"ShutdownLevel_LoadLDAPUsers"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_MetricsManager"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Pipeline"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Queue"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_CallbackRunner"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_HttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_DmcProxyHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Duo2FAHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - Shutdown complete in
1561.8 milliseconds
07-25-2018 21:03:58.322 +0000 INFO loader - All pipelines finished.

[sudosplunk]

Try looking for information in splunkd_stderr.log and splunkd_stdout.log. These log files contains some information about startup process.
PS: File path is same, /opt/splunkforwarder/var/log/splunk/

[mwdbhyat]

You can check your _internal logs to see if there are any issues during deployment. Does Splunk get restarted on the forwarder when its deployed? Are there any errors during restart?

[akadoya]

Does Splunk get restarted on the forwarder when its deployed?

No. I mentioned " it doesn't start app" but it's actually not starting up the forwarder process itself.

Are there any errors during restart?

I didn't see anything unusual in _internal index. Is there any particular component or log file name I should look into for deployment server behavior?