https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125351#M97710 <P>follow @martin_mueller advice and check the servers ulimit settings; usually if something works for a few minutes and then stops on *nix systems, indicates ulimit being too low. </P> Sat, 15 Nov 2014 10:07:42 GMT MuS 2014-11-15T10:07:42Z https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125349#M97708 <P>I have a ticket in with support but this may be faster.</P> <P>My intermediate forwarder is not working right. When I restart it, everything works for a few minutes then stops working. I have checked everything that I know to help.</P> <P>Please help with suggestions. 600 systems are down!!!</P> Sat, 15 Nov 2014 00:58:48 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125349#M97708 hartfoml 2014-11-15T00:58:48Z https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125350#M97709 <P>Usually there would be some indication of what's wrong in the IF's internal logs, especially splunkd.log.</P> Sat, 15 Nov 2014 09:55:21 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125350#M97709 martin_mueller 2014-11-15T09:55:21Z https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125351#M97710 <P>follow @martin_mueller advice and check the servers ulimit settings; usually if something works for a few minutes and then stops on *nix systems, indicates ulimit being too low. </P> Sat, 15 Nov 2014 10:07:42 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125351#M97710 MuS 2014-11-15T10:07:42Z https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125352#M97711 <P>Splunk Support was relatively quick to respond. Rajpal Bal got on the line and at my request she quickly setup a webex. we looked at SOS and could see that the tcpout on the Intermediate forwarder (IF) was full and the tcpin for the indexers was very low. on Thursday there was a mix up in DNS but this did not affect the IF until Splunk was restarted yesterday. Rajpal suggested and helped me to add the connection_host entry below to the inputs.conf to force Splunk to use IP and not look-up DNS names. we did this on both the IF and the indexers. it did not immediately resolve the issues but over a few hours the IF started its normal behavior and we can fix DNS on Monday.</P> <P>Thanks Rajpal, for fast, appropriate and extra effort in staying beyond work hours to solve this tricky problem</P> <P>in the inputs.conf that has the "splunktcp" stanza the "connection_host = ip" for app ports like below </P> <P><CODE>[splunktcp:://]</CODE><BR /> <CODE>connection_host = ip</CODE></P> Sat, 15 Nov 2014 12:59:58 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-troubleshoot-why-my-intermediate-forwarder-is-not-working/m-p/125352#M97711 hartfoml 2014-11-15T12:59:58Z