Getting Data In

Why our Universal Forwarder frequently stop forwarding logs?

dantimola
Communicator

Hi All,

Good Day, I have a problem with our universal forwarder, it frequently stops forwarding data. When the problem occur, my temporary resolution is to restart the forwarder and it will forward data again, however, the next day problem will occur again. It happen almost every day. What could be the solution here?

Universal Forwarder version: 6.2.6 (build 274160)

Thanks,
Dan

Tags (1)

dantimola
Communicator

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

0 Karma

dantimola
Communicator

Here's splunkd.log before the problem occurred.

alt text

0 Karma

skalliger
Motivator

Could you filter your internal events for any errors, please? Right now, it's hard to tell why the connection got interrupted.

Skalli

0 Karma

dantimola
Communicator

What does Cooked connection and Ping connection means beside of network error?

0 Karma

dantimola
Communicator

06-07-2017 17:11:07.135 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:11:12.923 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:12.923 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:11:42.875 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:12:02.718 +0800 WARN TcpOutputProc - Cooked connection to ip=IP:9997 timed out
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.164 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:12:07.180 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:00.189 +0800 WARN TcpOutputProc - Forwarding to indexer group group1 blocked for 100 seconds.
06-07-2017 17:13:07.193 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.209 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:07.224 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IP_8089_IP_pamapd02_508640B8-DCCC-43F7-BAEA-C19506B7C372
06-07-2017 17:13:12.372 +0800 WARN TcpOutputProc - Raw connection to ip=IP:9997 timed out
06-07-2017 17:13:12.372 +0800 INFO TcpOutputProc - Ping connection to idx=IP:9997 timed out. continuing connections
06-07-2017 17:13:23.105 +0800 INFO TcpOutputProc - Connected to idx=IP:9998
06-07-2017 17:17:51.067 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISDP-Root-CCBDPD02_logadm.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-07-2017 17:17:51.083 +0800 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\Program Files (x86)\CyberArk\Password Manager\Logs\ThirdParty\HP_pseudo-ISD VA-Root-CCBDPD02_secadmin.Debug.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

0 Karma

skalliger
Motivator

Did you check whether the UF also stops sending _internal logs? Please show us your splunkd.log and the metrics.log from the time the forwarder stopped sending logs.

0 Karma
Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...