Getting Data In

forwarder stopped sending to indexer but continues to send to 3rd party receiver

Log_wrangler
Builder

Any help on this is greatly appreciated.

I have a bunch of servers with UFs sending to a HF that is configured to send cooked data to splunk indexers and raw uncooked to a 3rd party receiver.

The flows have been running fine, but suddenly data to the indexers stopped.

I am looking for any suggestions to troubleshoot this.

Thank you

0 Karma
1 Solution

Log_wrangler
Builder

Thanks, did that and could see that the 3rd party receiving ip was blocking, so there must have been a build up that ultimately stopped the forwarder...

02-22-2018 16:44:56.119 -0600 INFO TcpOutputProc - Connected to idx=1.1.1.1:some_port
02-22-2018 16:53:11.041 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 10 seconds.
02-22-2018 16:53:21.024 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 20 seconds.
02-22-2018 16:54:09.009 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 40 seconds.

View solution in original post

0 Karma

Log_wrangler
Builder
  • should mention, that eventually everything stopped receiving, after a few minutes.
0 Karma

Log_wrangler
Builder

Thanks, did that and could see that the 3rd party receiving ip was blocking, so there must have been a build up that ultimately stopped the forwarder...

02-22-2018 16:44:56.119 -0600 INFO TcpOutputProc - Connected to idx=1.1.1.1:some_port
02-22-2018 16:53:11.041 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 10 seconds.
02-22-2018 16:53:21.024 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 20 seconds.
02-22-2018 16:54:09.009 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 40 seconds.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Check the logs.

0 Karma
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...