Getting Data In

Why our Universal Forwarder frequently stop forwarding logs?

dantimola
Communicator

Hi All,

Good Day, I have a problem with our universal forwarder, it frequently stops forwarding data. When the problem occur, my temporary resolution is to restart the forwarder and it will forward data again, however, the next day problem will occur again. It happen almost every day. What could be the solution here?

Universal Forwarder version: 6.2.6 (build 274160)

Thanks,
Dan

Tags (1)

dantimola
Communicator

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

0 Karma

dantimola
Communicator

Here's splunkd.log before the problem occurred.

alt text

0 Karma

skalliger
SplunkTrust
SplunkTrust

Could you filter your internal events for any errors, please? Right now, it's hard to tell why the connection got interrupted.

Skalli

0 Karma

dantimola
Communicator

What does Cooked connection and Ping connection means beside of network error?

0 Karma

dantimola
Communicator

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

0 Karma

skalliger
SplunkTrust
SplunkTrust

Did you check whether the UF also stops sending _internal logs? Please show us your splunkd.log and the metrics.log from the time the forwarder stopped sending logs.

0 Karma
Get Updates on the Splunk Community!

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...

What’s New in Splunk Cloud Platform 9.1.2308?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2308! Analysts can ...