We have a single data source from which we want to forward clone data to - splunk server 1(prod) and splunk server 2(qa).
The data seems to go to splunk server 1 fine but doesn't get forwarded to splunk server 2. We don't anything wrong in the log file too.
splunk list forward-server lists both the servers
outputs.conf (Windows Forwarder)
[tcpout] defaultGroup=awsprod,awsdev [tcpout:awsprod] server=<server1-ip>:9997 useACK = true [tcpout:awsdev] server=<server2-ip>:9997 useACK = true
As a work around, we have put a forward stanza on splunk server 1(prod) to forward data to splunk server 2(qa) and it seems to work fine.
When we try to forward data from other machines to server2 (qa), it seems to work fine.
Any suggestions are highly appreciated.
PS: More details on cloning and server details - qa/prod added.
Based on configuration which you have provided, this will clone data to both the Indexer (Server -1 and Server - 2). What you want to achieve, do you want to send data to both the indexer in load balance way (Not cloning of data) then answer provided by @Elsurion is correct with minor modification.
[tcpout] defaultGroup=awsprod [tcpout:awsprod] server=<server1-ip>:9997,<server2-ip>:9997 useACK = true
Our requirement is to clone data to both the servers. The servers are QA and Prod instances respectively.
The surprising part is data is not reaching QA and as a work around we have setup forwarding from Prod to QA.
In that case configuration which you have provided is correct and I am assuming you are not using
_TCP_ROUTING in your monitor stanza in inputs.conf
Can you please check from your UF to Server -2 network connectivity using telnet command
telnet Server_2_IP 9997 ?
Ok, can you please try to run below query on Server-2 (Indexer-2) so that we can check whether you are receiving data on Server-2 from UF or not
index=_internal host=Sever2 source=*metrics.log* group=per_host_thruput series=UF_FQDN
You can forward only to one destination that way, if you'd like to forward the data to two indexers, then you have to combine it.
[tcpout] defaultGroup=awsprod,awsdev [tcpout:awsprod] server=<server1-ip>:9997,server=<server2-ip>:9997 useACK = true
I assume you don't have Index replication enabled.
From the docs, if we give server list in comma separated fashion, the data will be load balanced between two receivers. Please confirm if my understanding is correct.
# Specify a target group made up of two receivers. In this case, the data will # be distributed using AutoLB between these two receivers. You can specify as # many receivers as you wish here. You can combine host name and IP if you # wish. # NOTE: Do not use this configuration with SplunkLightForwarder. [tcpout:group3] server=myhost.Splunk.com:9997,10.1.1.197:6666
Yes, forwarder will send data in a load balanced way and data will not be cloned if you are using below configuration:-
useACK = true
But if you want to clone could you try the below configuration in your outputs.conf
defaultGroup = awsprod
There is 99% chance you might have misconfigured forwarder.
on indexers search app look for the output of below query
If you get the data it means you have configure the forwarder properly. If you get the logs then look for errors in those logs.
Refer this link:
Also check the output at the forwarder cli in order to check the connectivity
telnet indexer-ip 8089 telnet indexer-ip 9997
Check if you have enabled forwarder receiving port 9997 on both indexers.
Also check if the monitor stanza that you have written is correct or not!
Let me know if this helps!!
Thanks for the note.
Here is the inputs.conf
What surprises us is that data is getting forwarded to one server. We will do telnet test and report our findings.
hey I faced the same problem while getting data in from on TCP
Everything was working fine. So the problem got solved by enabling IP forwarding on the server.
Refer this link, and let me know:
thanks mayurr98. telnet is connecting.
We are able to forward data to server 2 from
that is what surprises us.
We'll check the server side logs reg. forwarder.