We have a single data source from which we want to forward clone data to - splunk server 1(prod) and splunk server 2(qa).
The data seems to go to splunk server 1 fine but doesn't get forwarded to splunk server 2. We don't anything wrong in the log file too.
splunk list forward-server lists both the servers
outputs.conf (Windows Forwarder)
[tcpout] defaultGroup=awsprod,awsdev [tcpout:awsprod] server=<server1-ip>:9997 useACK = true [tcpout:awsdev] server=<server2-ip>:9997 useACK = true
As a work around, we have put a forward stanza on splunk server 1(prod) to forward data to splunk server 2(qa) and it seems to work fine.
When we try to forward data from other machines to server2 (qa), it seems to work fine.
Any suggestions are highly appreciated.
PS: More details on cloning and server details - qa/prod added.
There is 99% chance you might have misconfigured forwarder.
on indexers search app look for the output of below query
If you get the data it means you have configure the forwarder properly. If you get the logs then look for errors in those logs.
Refer this link:
Also check the output at the forwarder cli in order to check the connectivity
telnet indexer-ip 8089 telnet indexer-ip 9997
Check if you have enabled forwarder receiving port 9997 on both indexers.
Also check if the monitor stanza that you have written is correct or not!
Let me know if this helps!!
Thanks for the note.
Here is the inputs.conf
What surprises us is that data is getting forwarded to one server. We will do telnet test and report our findings.
thanks mayurr98. telnet is connecting.
We are able to forward data to server 2 from
that is what surprises us.
We'll check the server side logs reg. forwarder.
hey I faced the same problem while getting data in from on TCP
Everything was working fine. So the problem got solved by enabling IP forwarding on the server.
Refer this link, and let me know:
You can forward only to one destination that way, if you'd like to forward the data to two indexers, then you have to combine it.
[tcpout] defaultGroup=awsprod,awsdev [tcpout:awsprod] server=<server1-ip>:9997,server=<server2-ip>:9997 useACK = true
I assume you don't have Index replication enabled.
From the docs, if we give server list in comma separated fashion, the data will be load balanced between two receivers. Please confirm if my understanding is correct.
# Specify a target group made up of two receivers. In this case, the data will # be distributed using AutoLB between these two receivers. You can specify as # many receivers as you wish here. You can combine host name and IP if you # wish. # NOTE: Do not use this configuration with SplunkLightForwarder. [tcpout:group3] server=myhost.Splunk.com:9997,10.1.1.197:6666