Splunk Forwarding doesn't forward data

[ykpramodhcbt]

We have a single data source from which we want to forward clone data to - splunk server 1(prod) and splunk server 2(qa).

The data seems to go to splunk server 1 fine but doesn't get forwarded to splunk server 2. We don't anything wrong in the log file too.

splunk list forward-server lists both the servers

outputs.conf (Windows Forwarder)

[tcpout]
defaultGroup=awsprod,awsdev

[tcpout:awsprod]
server=<server1-ip>:9997
useACK = true

[tcpout:awsdev]
server=<server2-ip>:9997
useACK = true

As a work around, we have put a forward stanza on splunk server 1(prod) to forward data to splunk server 2(qa) and it seems to work fine.
When we try to forward data from other machines to server2 (qa), it seems to work fine.

Any suggestions are highly appreciated.

PS: More details on cloning and server details - qa/prod added.

[harsmarvania57]

Based on configuration which you have provided, this will clone data to both the Indexer (Server -1 and Server - 2). What you want to achieve, do you want to send data to both the indexer in load balance way (Not cloning of data) then answer provided by @Elsurion is correct with minor modification.

[tcpout]
defaultGroup=awsprod

[tcpout:awsprod]
server=<server1-ip>:9997,<server2-ip>:9997
useACK = true

[ykpramodhcbt]

Our requirement is to clone data to both the servers. The servers are QA and Prod instances respectively.

The surprising part is data is not reaching QA and as a work around we have setup forwarding from Prod to QA.

[harsmarvania57]

In that case configuration which you have provided is correct and I am assuming you are not using _TCP_ROUTING in your monitor stanza in inputs.conf

Can you please check from your UF to Server -2 network connectivity using telnet command telnet Server_2_IP 9997 ?

[ykpramodhcbt]

Thanks for your note.

We are not using _TCP_ROUTING

telnet Server2 9997 is working fine.

[harsmarvania57]

Ok, can you please try to run below query on Server-2 (Indexer-2) so that we can check whether you are receiving data on Server-2 from UF or not

index=_internal host=Sever2 source=*metrics.log* group=per_host_thruput series=UF_FQDN 

[Elsurion]

You can forward only to one destination that way, if you'd like to forward the data to two indexers, then you have to combine it.

 [tcpout]
 defaultGroup=awsprod,awsdev

 [tcpout:awsprod]
 server=<server1-ip>:9997,server=<server2-ip>:9997
 useACK = true

I assume you don't have Index replication enabled.

[ykpramodhcbt]

From the docs, if we give server list in comma separated fashion, the data will be load balanced between two receivers. Please confirm if my understanding is correct.

# Specify a target group made up of two receivers.  In this case, the data will
# be distributed using AutoLB between these two receivers.  You can specify as
# many receivers as you wish here. You can combine host name and IP if you
# wish.
# NOTE: Do not use this configuration with SplunkLightForwarder.

[tcpout:group3]
server=myhost.Splunk.com:9997,10.1.1.197:6666

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Outputsconf

[nikita_p]

Yes, forwarder will send data in a load balanced way and data will not be cloned if you are using below configuration:-
[tcpout]
defaultGroup=awsprod,awsdev

[tcpout:awsprod]
server=:9997,server=:9997
useACK = true

But if you want to clone could you try the below configuration in your outputs.conf
[tcpout]
defaultGroup = awsprod

[tcpout-server://:9997]

[tcpout-server://1:9997]

[nikita_p]

Hi,
Sorry there was some typo in my outputs.conf
[tcpout]
defaultGroup = awsprod

[tcpout-server://server1-ip:9997]
[tcpout-server://server2-ip:9997]

[ykpramodhcbt]

We will try this and we will update you.

[ykpramodhcbt]

Sorry if we have not added sufficient details earlier. We wish to clone data to both the servers as they are QA and Prod respectively.

[mayurr98]

There is 99% chance you might have misconfigured forwarder.
on indexers search app look for the output of below query

index=_internal host=forwarder

If you get the data it means you have configure the forwarder properly. If you get the logs then look for errors in those logs.
Refer this link:
http://docs.splunk.com/Documentation/Forwarder/7.0.1/Forwarder/Configuretheuniversalforwarder

Also check the output at the forwarder cli in order to check the connectivity

telnet indexer-ip 8089
telnet indexer-ip 9997

Check if you have enabled forwarder receiving port 9997 on both indexers.
Also check if the monitor stanza that you have written is correct or not!
Let me know if this helps!!

[ykpramodhcbt]

Hi mayurr98,

Thanks for the note.

Here is the inputs.conf

[monitor://d:\Carbynetech.csv]
disabled=false
index=indexname

What surprises us is that data is getting forwarded to one server. We will do telnet test and report our findings.

regards
Pramodh

[ykpramodhcbt]

Hi Mayurr98, the tcp connection from server2 to destination splunk server on ports 8089 and 9997 are working as expected.

[mayurr98]

hey I faced the same problem while getting data in from on TCP
Everything was working fine. So the problem got solved by enabling IP forwarding on the server.
Refer this link, and let me know:
http://www.ducea.com/2006/08/01/how-to-enable-ip-forwarding-in-linux/

[mayurr98]

Oh then mostly the problem is of connectivity do check telnet test.
Also Check for forwarder logs on second server
Are they populating?

[ykpramodhcbt]

thanks mayurr98. telnet is connecting.

We are able to forward data to server 2 from

1. another forwarder on another machine
2. server 1 => forwarding the information

that is what surprises us.

We'll check the server side logs reg. forwarder.