Why did forwarder shutdown after start?

[pbnl]

hi again 🙂

after upgrading our 26 linux universal forwarders from 7.x to 8.2.5, one of them will not run anymore.
it immediately shuts down itself after start.
splunkd.log shows nothing special for me. here some lines:

03-30-2022 13:03:03.587 +0200 INFO WatchedFile [110115 tailreader0] - File too small to check seek
crc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunk_instrumentation_cloud.log'.
03-30-2022 13:03:04.440 +0200 INFO AutoLoadBalancedConnectionStrategy [110109 TcpOutEloop] - Connected to idx=172.17.1.91:9997, pset=0, reuse=0. using ACK.
03-30-2022 13:03:05.997 +0200 INFO loader [110063 HTTPDispatch] - Shutdown HTTPDispatchThread
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - Shutting down splunkd
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - shutting down level "ShutdownLevel_Begin"
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - shutting down level "ShutdownLevel_NoahHealthReport"

any ideas?
thanks...

[gcusello]

Hi @pbnl,

did you checked the available memory?

did you checked if the old installation is always active?

Ciao.

Giuseppe

[pbnl · Accepted solution]

hi @gcusello ,

thanks for the fast answer, but that wasn't the problem.

i used this app adapted to my needs to upgrade my forwarders. for some reason, on the not running forwarder, the script was always started, everytime splunkd was started. it stopped splunkd and exited.
removed the app, and it's running again.

strange, but hey, it's a computer 😉

[mygoalfinder]

Hi I am having the exact same issue. I am attempting to use that script to upgrade my infrastructure, but it stops immediately after assigning it to the clients. Why is that? What could be a workaround for this? 

I also tested it manually. The script works fine manually, but refuses to upgrade automatically. It just shuts down.

[gcusello]

Hi @mygoalfinder ,

it isn't a good idea to add a question to another, especially if solved because none will answer you: open a new case to be sure to have more answers.

Anyway, did you stopped the running splunk before upgrading?

Ciao.

Giuseppe

[mygoalfinder]

Yes the script stops the forwarder, but does not start back up. Using the exact same script that was listed above in the link the other user provided.

[gcusello]

Hi @pbnl,

good for you if you solved your problem, please accept the answer for the other people of Community, otherwise tell me how can I help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[pbnl]

ok. i will do.

and indeed, you can help me with another issue. but in a new thread 😉