Getting Data In

forwarder stopped sending to indexer but continues to send to 3rd party receiver

Log_wrangler
Builder

Any help on this is greatly appreciated.

I have a bunch of servers with UFs sending to a HF that is configured to send cooked data to splunk indexers and raw uncooked to a 3rd party receiver.

The flows have been running fine, but suddenly data to the indexers stopped.

I am looking for any suggestions to troubleshoot this.

Thank you

0 Karma
1 Solution

Log_wrangler
Builder

Thanks, did that and could see that the 3rd party receiving ip was blocking, so there must have been a build up that ultimately stopped the forwarder...

02-22-2018 16:44:56.119 -0600 INFO TcpOutputProc - Connected to idx=1.1.1.1:some_port
02-22-2018 16:53:11.041 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 10 seconds.
02-22-2018 16:53:21.024 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 20 seconds.
02-22-2018 16:54:09.009 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 40 seconds.

View solution in original post

0 Karma

Log_wrangler
Builder
  • should mention, that eventually everything stopped receiving, after a few minutes.
0 Karma

Log_wrangler
Builder

Thanks, did that and could see that the 3rd party receiving ip was blocking, so there must have been a build up that ultimately stopped the forwarder...

02-22-2018 16:44:56.119 -0600 INFO TcpOutputProc - Connected to idx=1.1.1.1:some_port
02-22-2018 16:53:11.041 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 10 seconds.
02-22-2018 16:53:21.024 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 20 seconds.
02-22-2018 16:54:09.009 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 40 seconds.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Check the logs.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...