Deployment Architecture

restartSplunkd doesn't restart Splunkd

akadoya
Explorer

Hi, I am deploying apps from deployment server.
Server classes having restartSplunkd=1 gets to stop when I deploy new apps but it doesn't start app.

How does the deployment server send a request to remote client?
is there any way to see what exactly going wrong here?

Tags (1)

sudosplunk
Motivator

After checking splunkd.log for errors, also check for any typo's in your configuration files. Splunk doesn't start if it encounters any typo's or mistakes in critical configuration settings. You can use btool to troubleshoot your configuration files. HTH!

0 Karma

akadoya
Explorer

If I restart the splunkd manually on the client side, I can start it up so I don't think it's typo or any configuration error.

This is the client that just got new app deployed. it received signal from deployment server and shutdown itself successfully.
it just doesn't come back up again.
VVV

tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log
07-25-2018 21:03:57.382 +0000 INFO  ShutdownHandler - shutting down level

"ShutdownLevel_LoadLDAPUsers"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_MetricsManager"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Pipeline"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Queue"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_CallbackRunner"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_HttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_DmcProxyHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Duo2FAHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - Shutdown complete in
1561.8 milliseconds
07-25-2018 21:03:58.322 +0000 INFO loader - All pipelines finished.

0 Karma

sudosplunk
Motivator

Try looking for information in splunkd_stderr.log and splunkd_stdout.log. These log files contains some information about startup process.
PS: File path is same, /opt/splunkforwarder/var/log/splunk/

0 Karma

mwdbhyat
Builder

You can check your _internal logs to see if there are any issues during deployment. Does Splunk get restarted on the forwarder when its deployed? Are there any errors during restart?

0 Karma

akadoya
Explorer

Does Splunk get restarted on the forwarder when its deployed?

No. I mentioned " it doesn't start app" but it's actually not starting up the forwarder process itself.

Are there any errors during restart?

I didn't see anything unusual in _internal index. Is there any particular component or log file name I should look into for deployment server behavior?

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...