Getting Data In

Why our Universal Forwarder frequently stop forwarding logs?

dantimola
Communicator

Hi All,

Good Day, I have a problem with our universal forwarder, it frequently stops forwarding data. When the problem occur, my temporary resolution is to restart the forwarder and it will forward data again, however, the next day problem will occur again. It happen almost every day. What could be the solution here?

Universal Forwarder version: 6.2.6 (build 274160)

Thanks,
Dan

Tags (1)

dantimola
Communicator

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

0 Karma

dantimola
Communicator

Here's splunkd.log before the problem occurred.

alt text

0 Karma

skalliger
Motivator

Could you filter your internal events for any errors, please? Right now, it's hard to tell why the connection got interrupted.

Skalli

0 Karma

dantimola
Communicator

What does Cooked connection and Ping connection means beside of network error?

0 Karma

dantimola
Communicator

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

0 Karma

skalliger
Motivator

Did you check whether the UF also stops sending _internal logs? Please show us your splunkd.log and the metrics.log from the time the forwarder stopped sending logs.

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...