topic Re: forwarder shutdown after start in Installation

Why did forwarder shutdown after start?

pbnl — Wed, 30 Mar 2022 17:00:32 GMT

hi again 🙂

after upgrading our 26 linux universal forwarders from 7.x to 8.2.5, one of them will not run anymore.
it immediately shuts down itself after start.
splunkd.log shows nothing special for me. here some lines:

03-30-2022 13:03:03.587 +0200 INFO WatchedFile [110115 tailreader0] - File too small to check seek
crc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunk_instrumentation_cloud.log'.
03-30-2022 13:03:04.440 +0200 INFO AutoLoadBalancedConnectionStrategy [110109 TcpOutEloop] - Connected to idx=172.17.1.91:9997, pset=0, reuse=0. using ACK.
03-30-2022 13:03:05.997 +0200 INFO loader [110063 HTTPDispatch] - Shutdown HTTPDispatchThread
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - Shutting down splunkd
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - shutting down level "ShutdownLevel_Begin"
03-30-2022 13:03:06.031 +0200 INFO ShutdownHandler [110076 Shutdown] - shutting down level "ShutdownLevel_NoahHealthReport"

any ideas?
thanks...

Re: forwarder shutdown after start

gcusello — Wed, 30 Mar 2022 11:50:35 GMT

Hi @pbnl,

did you checked the available memory?

did you checked if the old installation is always active?

Ciao.

Giuseppe

Re: forwarder shutdown after start

pbnl — Wed, 30 Mar 2022 12:49:44 GMT

hi @gcusello ,

thanks for the fast answer, but that wasn't the problem.

i used this app adapted to my needs to upgrade my forwarders. for some reason, on the not running forwarder, the script was always started, everytime splunkd was started. it stopped splunkd and exited.
removed the app, and it's running again.

strange, but hey, it's a computer 😉

Re: forwarder shutdown after start

gcusello — Thu, 31 Mar 2022 12:14:03 GMT

Hi @pbnl,

good for you if you solved your problem, please accept the answer for the other people of Community, otherwise tell me how can I help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: forwarder shutdown after start

pbnl — Thu, 31 Mar 2022 12:33:40 GMT

ok. i will do.

and indeed, you can help me with another issue. but in a new thread 😉

Re: forwarder shutdown after start

mygoalfinder — Tue, 21 Mar 2023 17:00:33 GMT

Hi I am having the exact same issue. I am attempting to use that script to upgrade my infrastructure, but it stops immediately after assigning it to the clients. Why is that? What could be a workaround for this?

I also tested it manually. The script works fine manually, but refuses to upgrade automatically. It just shuts down.

Re: forwarder shutdown after start

gcusello — Tue, 21 Mar 2023 17:23:48 GMT

Hi @mygoalfinder ,

it isn't a good idea to add a question to another, especially if solved because none will answer you: open a new case to be sure to have more answers.

Anyway, did you stopped the running splunk before upgrading?

Ciao.

Giuseppe

Re: forwarder shutdown after start

mygoalfinder — Tue, 21 Mar 2023 17:26:36 GMT

Yes the script stops the forwarder, but does not start back up. Using the exact same script that was listed above in the link the other user provided.