Hi,
I have set up 4 forwarders to communicate with my indexer. I already executed add forward-server and set deploy-poll and created the outputs.conf from deployment-apps. On the forwader management, I can only see 3 instead of 4 forwarders available. I checked the _internal logs and got the following errors related to ShutdownHandler and TcpOutputProc and TcpOutputFd. Below are sample logs that I got:
03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_ArchiveAndOneshot"
host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd
3/3/20
1:19:44.312 PM
03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput"
host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd
3/3/20
1:19:44.312 PM
03-03-2020 13:19:44.312 -0500 INFO TcpInputProc - TCP connection cleanup complete
host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd
3/3/20
1:19:44.312 PM
03-03-2020 13:19:44.312 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_Scheduler"
host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd
3/2/20
9:16:43.406 AM
03-02-2020 09:16:43.406 -0500 WARN TcpOutputFd - Connect to xxx:9997 failed. Connection refused
host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd
3/2/20
9:16:43.042 AM
03-02-2020 09:16:43.042 -0500 INFO TcpOutputProc - Connection to xxx:9997 closed. Connection closed by server.
host = xxx source = /opt/splunkforwarder/var/log/splunk/splunkd.logsourcetype = splunkd
My guess is is being force to shutdon by the server or admin. Tho i am not sure. Does anyone experienced this? Any help is appreciated.
