Why is Splunk not receiving on splunktcp 9997?

[khagan]

I have a Heavy Forwarder set to forward load balanced data to two Splunk indexers on 9997.

When I enable receiving on the indexers (via Settings -> Forwarding and Receiving -> Configure Receiving), no data is showing up.

Examining the splunkd.log on the forwarder and indexers shows the connection is being made with no errors.

If I enable a Data Input to listen to port 9997, I can see data showing up in the index, albeit cooked data, which isn't readable - so the data is making it to the Splunk server, but just not showing up when I configure it to receive from another Splunk.

What might be causing this?

[rkirkw]

For more troubleshooting info on receiving try this command:

splunk list inputstatus

This gives you a listing of every way an indexer is receiving data from each host and when

Hope this helps!!

[somesoni2]

Can you provide the content of outputs.conf from your Heavy Forwarder? After enabling receiving on Forwarder, you would configure your forwarder to send data to those indexers on port 9997.

[khagan]

[tcpout]
defaultGroup=LB_peers

[tcpout:LB_peers]
server=xx.xx.xx.xx:9997,xx.xx.xx.xx:9997

The Forwarder is already configured to send to Splunk on 9997.

[janderson19]

I'm not really sure how to receive from a forwarder over Splunk web, but if you go into inputs.conf you can make sure you're receiving over splunktcp. If not, you can change it to Splunktcp.

[khagan]

Inputs.conf has it receiving on [splunktcp:9997]

[janderson19]

Correct format is [splunktcp://:9997]

http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/inputsconf

[khagan]

Yes, sorry, that is the format it is in.