Installation

Why did forwarder shutdown after start?

pbnl
Path Finder

hi again 🙂

after upgrading our 26 linux universal forwarders from 7.x to 8.2.5, one of them will not run anymore.
it immediately shuts down itself after start.
splunkd.log shows nothing special for me. here some lines:

03-30-2022 13:03:03.587 +0200 INFO WatchedFile [110115 tailreader0] - File too small to check seek
crc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunk_instrumentation_cloud.log'.
03-30-2022 13:03:04.440 +0200 INFO AutoLoadBalancedConnectionStrategy [110109 TcpOutEloop] - Connected to idx=172.17.1.91:9997, pset=0, reuse=0. using ACK.
03-30-2022 13:03:05.997 +0200 INFO loader [110063 HTTPDispatch] - Shutdown HTTPDispatchThread
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - Shutting down splunkd
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - shutting down level "ShutdownLevel_Begin"
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - shutting down level "ShutdownLevel_NoahHealthReport"

any ideas?
thanks...

Labels (3)
0 Karma
1 Solution

pbnl
Path Finder

hi @gcusello ,

thanks for the fast answer, but that wasn't the problem.

i used this app adapted to my needs to upgrade my forwarders. for some reason, on the not running forwarder, the script was always started, everytime splunkd was started. it stopped splunkd and exited.
removed the app, and it's running again.

strange, but hey, it's a computer 😉

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @pbnl,

did you checked the available memory?

did you checked if the old installation is always active?

Ciao.

Giuseppe

0 Karma

pbnl
Path Finder

hi @gcusello ,

thanks for the fast answer, but that wasn't the problem.

i used this app adapted to my needs to upgrade my forwarders. for some reason, on the not running forwarder, the script was always started, everytime splunkd was started. it stopped splunkd and exited.
removed the app, and it's running again.

strange, but hey, it's a computer 😉

0 Karma

mygoalfinder
Explorer

Hi I am having the exact same issue. I am attempting to use that script to upgrade my infrastructure, but it stops immediately after assigning it to the clients. Why is that? What could be a workaround for this? 

 

I also tested it manually. The script works fine manually, but refuses to upgrade automatically. It just shuts down.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mygoalfinder ,

it isn't a good idea to add a question to another, especially if solved because none will answer you: open a new case to be sure to have more answers.

Anyway, did you stopped the running splunk before upgrading?

Ciao.

Giuseppe

0 Karma

mygoalfinder
Explorer

Yes the script stops the forwarder, but does not start back up. Using the exact same script that was listed above in the link the other user provided.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @pbnl,

good for you if you solved your problem, please accept the answer for the other people of Community, otherwise tell me how can I help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

pbnl
Path Finder

ok. i will do.

and indeed, you can help me with another issue. but in a new thread 😉

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...