Getting Data In

Universal Forwarder to Universal Forwarder to Indexer

vial8
Engager

I am currently configuring systems to forward data to splunk, but I have hit a wall with the Universal forwarder configuration.

My setup looks as follows:

I have my main indexer in a DC ( Let's call it head)
Then I have 1 main forwarder in another DC which forwards all the data to head. ( Let’s call this forward1 )
All my servers in the same DC as foward1 sends it data to forward1, and in turn forward1 needs to send it to head.

To further complicate the picture I have another DC.
I have the same scenario there.
A forwarder to collect all the data for that dc ( Let’s call it forward2)

it will then send all the data from forward2 to forward1 which in turn will send it to head.

My problem is this.

I have forward1 up and running sending data to head.
I tell forward1 to listen on port 9997. All good.

I then start sending data to it from a server with a forwarder on I get the following error:

on the server I see:
04-29-2013 14:47:58.033 +0200 WARN TcpOutputProc - Cooked connection to ip=10.13.1.24:9997 timed out

On forward1 I get this:
04-29-2013 14:40:01.643 +0200 INFO TcpInputProc - Connection in raw mode from src=10.13.2.3:53381

I have exhausted all the resources but am getting nowhere. Do you have any idea what can be wrong here ?

My set-up looks like this.

App server that needs to send data to Forward1
./splunk list forward-server

Active forwards:
None
Configured but inactive forwards:
10.13.1.24:9997 <-- Forward1 server

My biggest concern is that it is not active, and I cannot figure out why.
No firewall issues , can ping it and telnet to it.

Forward1

splunk list forward-server

Active forwards: 10.0.64.120:9997 <-- Head server

Configured but inactive forwards:
None

splunk list tcp

Splunk is listening for data on ports:
9997 for data from any host

I am at my wits end here, any help will be greatly appreciated. I have searched the knowledge base came across a lot of similar cases, but none of their solutions fixed my problem.

Tags (2)
0 Karma
1 Solution

Ayn
Legend

You've chosen the wrong type of TCP input on forward1. You've got a raw TCP input there on port 9997, but what you really want is a receiving port that is used specifically for receiving cooked data from other Splunk instances - in the manager, it's listed under the "Forwarding and receiving" section" rather than the "Data inputs" section.

More info on setting up receiving, and generally deploying Splunk in a distributed architecture, can be found here for instance:
http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Setupforwardingandreceiving
http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Enableareceiver

View solution in original post

Ayn
Legend

You've chosen the wrong type of TCP input on forward1. You've got a raw TCP input there on port 9997, but what you really want is a receiving port that is used specifically for receiving cooked data from other Splunk instances - in the manager, it's listed under the "Forwarding and receiving" section" rather than the "Data inputs" section.

More info on setting up receiving, and generally deploying Splunk in a distributed architecture, can be found here for instance:
http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Setupforwardingandreceiving
http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Enableareceiver

Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...