topic Why our Universal Forwarder frequently stop forwarding logs? in Getting Data In

Why our Universal Forwarder frequently stop forwarding logs?

dantimola — Wed, 07 Jun 2017 03:44:29 GMT

Hi All,

Good Day, I have a problem with our universal forwarder, it frequently stops forwarding data. When the problem occur, my temporary resolution is to restart the forwarder and it will forward data again, however, the next day problem will occur again. It happen almost every day. What could be the solution here?

Universal Forwarder version: 6.2.6 (build 274160)

Thanks,
Dan

Re: Why our Universal Forwarder frequently stop forwarding logs?

skalliger — Wed, 07 Jun 2017 07:04:14 GMT

Did you check whether the UF also stops sending _internal logs? Please show us your splunkd.log and the metrics.log from the time the forwarder stopped sending logs.

Re: Why our Universal Forwarder frequently stop forwarding logs?

dantimola — Wed, 07 Jun 2017 07:11:11 GMT

Here's splunkd.log before the problem occurred.

Re: Why our Universal Forwarder frequently stop forwarding logs?

skalliger — Wed, 07 Jun 2017 08:22:46 GMT

Could you filter your internal events for any errors, please? Right now, it's hard to tell why the connection got interrupted.

Skalli

Re: Why our Universal Forwarder frequently stop forwarding logs?

dantimola — Tue, 29 Sep 2020 14:19:38 GMT

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Re: Why our Universal Forwarder frequently stop forwarding logs?

dantimola — Wed, 07 Jun 2017 09:27:48 GMT

What does Cooked connection and Ping connection means beside of network error?

Re: Why our Universal Forwarder frequently stop forwarding logs?

dantimola — Tue, 29 Sep 2020 14:19:41 GMT

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.