https://community.splunk.com/t5/Getting-Data-In/Splunk-Forwarder-ShutdownHandler/m-p/487486#M83456 <P>Hi,</P> <P>I have set up 4 forwarders to communicate with my indexer. I already executed add forward-server and set deploy-poll and created the outputs.conf from deployment-apps. On the forwader management, I can only see 3 instead of 4 forwarders available. I checked the _internal logs and got the following errors related to ShutdownHandler and TcpOutputProc and TcpOutputFd. Below are sample logs that I got:</P> <P>03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_ArchiveAndOneshot"<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd<BR /> 3/3/20<BR /> 1:19:44.312 PM<BR /><BR /> 03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput"<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd<BR /> 3/3/20<BR /> 1:19:44.312 PM<BR /><BR /> 03-03-2020 13:19:44.312 -0500 INFO TcpInputProc - TCP connection cleanup complete<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd<BR /> 3/3/20<BR /> 1:19:44.312 PM<BR /><BR /> 03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_Scheduler"<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd </P> <P>3/2/20<BR /> 9:16:43.406 AM<BR /><BR /> 03-02-2020 09:16:43.406 -0500 WARN TcpOutputFd - Connect to xxx:9997 failed. Connection refused<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd<BR /> 3/2/20<BR /> 9:16:43.042 AM<BR /><BR /> 03-02-2020 09:16:43.042 -0500 INFO TcpOutputProc - Connection to xxx:9997 closed. Connection closed by server.<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd</P> <P>My guess is is being force to shutdon by the server or admin. Tho i am not sure. Does anyone experienced this? Any help is appreciated.</P> Wed, 30 Sep 2020 04:29:38 GMT ptrckjncbngn 2020-09-30T04:29:38Z https://community.splunk.com/t5/Getting-Data-In/Splunk-Forwarder-ShutdownHandler/m-p/487486#M83456 <P>Hi,</P> <P>I have set up 4 forwarders to communicate with my indexer. I already executed add forward-server and set deploy-poll and created the outputs.conf from deployment-apps. On the forwader management, I can only see 3 instead of 4 forwarders available. I checked the _internal logs and got the following errors related to ShutdownHandler and TcpOutputProc and TcpOutputFd. Below are sample logs that I got:</P> <P>03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_ArchiveAndOneshot"<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd<BR /> 3/3/20<BR /> 1:19:44.312 PM<BR /><BR /> 03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput"<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd<BR /> 3/3/20<BR /> 1:19:44.312 PM<BR /><BR /> 03-03-2020 13:19:44.312 -0500 INFO TcpInputProc - TCP connection cleanup complete<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd<BR /> 3/3/20<BR /> 1:19:44.312 PM<BR /><BR /> 03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_Scheduler"<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd </P> <P>3/2/20<BR /> 9:16:43.406 AM<BR /><BR /> 03-02-2020 09:16:43.406 -0500 WARN TcpOutputFd - Connect to xxx:9997 failed. Connection refused<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd<BR /> 3/2/20<BR /> 9:16:43.042 AM<BR /><BR /> 03-02-2020 09:16:43.042 -0500 INFO TcpOutputProc - Connection to xxx:9997 closed. Connection closed by server.<BR /> host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd</P> <P>My guess is is being force to shutdon by the server or admin. Tho i am not sure. Does anyone experienced this? Any help is appreciated.</P> Wed, 30 Sep 2020 04:29:38 GMT https://community.splunk.com/t5/Getting-Data-In/Splunk-Forwarder-ShutdownHandler/m-p/487486#M83456 ptrckjncbngn 2020-09-30T04:29:38Z