Why our Universal Forwarder frequently stop forwarding logs?

[dantimola]

Hi All,

Good Day, I have a problem with our universal forwarder, it frequently stops forwarding data. When the problem occur, my temporary resolution is to restart the forwarder and it will forward data again, however, the next day problem will occur again. It happen almost every day. What could be the solution here?

Universal Forwarder version: 6.2.6 (build 274160)

Thanks,
Dan

[dantimola]

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

[dantimola]

Here's splunkd.log before the problem occurred.

[skalliger]

Could you filter your internal events for any errors, please? Right now, it's hard to tell why the connection got interrupted.

Skalli

[dantimola]

What does Cooked connection and Ping connection means beside of network error?

[dantimola]

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

[skalliger]

Did you check whether the UF also stops sending _internal logs? Please show us your splunkd.log and the metrics.log from the time the forwarder stopped sending logs.