Deployment Architecture

restartSplunkd doesn't restart Splunkd

akadoya
Explorer

Hi, I am deploying apps from deployment server.
Server classes having restartSplunkd=1 gets to stop when I deploy new apps but it doesn't start app.

How does the deployment server send a request to remote client?
is there any way to see what exactly going wrong here?

Tags (1)

sudosplunk
Motivator

After checking splunkd.log for errors, also check for any typo's in your configuration files. Splunk doesn't start if it encounters any typo's or mistakes in critical configuration settings. You can use btool to troubleshoot your configuration files. HTH!

0 Karma

akadoya
Explorer

If I restart the splunkd manually on the client side, I can start it up so I don't think it's typo or any configuration error.

This is the client that just got new app deployed. it received signal from deployment server and shutdown itself successfully.
it just doesn't come back up again.
VVV

tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log
07-25-2018 21:03:57.382 +0000 INFO  ShutdownHandler - shutting down level

"ShutdownLevel_LoadLDAPUsers"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_MetricsManager"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Pipeline"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Queue"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_CallbackRunner"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_HttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_DmcProxyHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Duo2FAHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - Shutdown complete in
1561.8 milliseconds
07-25-2018 21:03:58.322 +0000 INFO loader - All pipelines finished.

0 Karma

sudosplunk
Motivator

Try looking for information in splunkd_stderr.log and splunkd_stdout.log. These log files contains some information about startup process.
PS: File path is same, /opt/splunkforwarder/var/log/splunk/

0 Karma

mwdbhyat
Builder

You can check your _internal logs to see if there are any issues during deployment. Does Splunk get restarted on the forwarder when its deployed? Are there any errors during restart?

0 Karma

akadoya
Explorer

Does Splunk get restarted on the forwarder when its deployed?

No. I mentioned " it doesn't start app" but it's actually not starting up the forwarder process itself.

Are there any errors during restart?

I didn't see anything unusual in _internal index. Is there any particular component or log file name I should look into for deployment server behavior?

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...