Universal Forwarder to Universal Forwarder to Indexer

[vial8]

I am currently configuring systems to forward data to splunk, but I have hit a wall with the Universal forwarder configuration.

My setup looks as follows:

I have my main indexer in a DC ( Let's call it head)
Then I have 1 main forwarder in another DC which forwards all the data to head. ( Let’s call this forward1 )
All my servers in the same DC as foward1 sends it data to forward1, and in turn forward1 needs to send it to head.

To further complicate the picture I have another DC.
I have the same scenario there.
A forwarder to collect all the data for that dc ( Let’s call it forward2)

it will then send all the data from forward2 to forward1 which in turn will send it to head.

My problem is this.

I have forward1 up and running sending data to head.
I tell forward1 to listen on port 9997. All good.

I then start sending data to it from a server with a forwarder on I get the following error:

on the server I see:
04-29-2013 14:47:58.033 +0200 WARN TcpOutputProc - Cooked connection to ip=10.13.1.24:9997 timed out

On forward1 I get this:
04-29-2013 14:40:01.643 +0200 INFO TcpInputProc - Connection in raw mode from src=10.13.2.3:53381

I have exhausted all the resources but am getting nowhere. Do you have any idea what can be wrong here ?

My set-up looks like this.

App server that needs to send data to Forward1
./splunk list forward-server

Active forwards:
None
Configured but inactive forwards:
10.13.1.24:9997 <-- Forward1 server

My biggest concern is that it is not active, and I cannot figure out why.
No firewall issues , can ping it and telnet to it.

Forward1

splunk list forward-server

Active forwards: 10.0.64.120:9997 <-- Head server

Configured but inactive forwards:
None

splunk list tcp

Splunk is listening for data on ports:
9997 for data from any host

I am at my wits end here, any help will be greatly appreciated. I have searched the knowledge base came across a lot of similar cases, but none of their solutions fixed my problem.

[Ayn · Accepted solution]

You've chosen the wrong type of TCP input on forward1. You've got a raw TCP input there on port 9997, but what you really want is a receiving port that is used specifically for receiving cooked data from other Splunk instances - in the manager, it's listed under the "Forwarding and receiving" section" rather than the "Data inputs" section.

More info on setting up receiving, and generally deploying Splunk in a distributed architecture, can be found here for instance:
http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Setupforwardingandreceiving
http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Enableareceiver