Getting Data In
Highlighted

Splunk Forwarding doesn't forward data

Path Finder

We have a single data source from which we want to forward clone data to - splunk server 1(prod) and splunk server 2(qa).

The data seems to go to splunk server 1 fine but doesn't get forwarded to splunk server 2. We don't anything wrong in the log file too.

splunk list forward-server lists both the servers

outputs.conf (Windows Forwarder)

[tcpout]
defaultGroup=awsprod,awsdev

[tcpout:awsprod]
server=<server1-ip>:9997
useACK = true

[tcpout:awsdev]
server=<server2-ip>:9997
useACK = true

As a work around, we have put a forward stanza on splunk server 1(prod) to forward data to splunk server 2(qa) and it seems to work fine.
When we try to forward data from other machines to server2 (qa), it seems to work fine.

Any suggestions are highly appreciated.

PS: More details on cloning and server details - qa/prod added.

0 Karma
Highlighted

Re: Splunk Forwarding doesn't forward data

SplunkTrust
SplunkTrust

There is 99% chance you might have misconfigured forwarder.
on indexers search app look for the output of below query

index=_internal host=forwarder

If you get the data it means you have configure the forwarder properly. If you get the logs then look for errors in those logs.
Refer this link:
http://docs.splunk.com/Documentation/Forwarder/7.0.1/Forwarder/Configuretheuniversalforwarder

Also check the output at the forwarder cli in order to check the connectivity

telnet indexer-ip 8089
telnet indexer-ip 9997

Check if you have enabled forwarder receiving port 9997 on both indexers.
Also check if the monitor stanza that you have written is correct or not!
Let me know if this helps!!

0 Karma
Highlighted

Re: Splunk Forwarding doesn't forward data

Path Finder

Hi mayurr98,

Thanks for the note.

Here is the inputs.conf

[monitor://d:\Carbynetech.csv]
disabled=false
index=indexname

What surprises us is that data is getting forwarded to one server. We will do telnet test and report our findings.

regards
Pramodh

0 Karma
Highlighted

Re: Splunk Forwarding doesn't forward data

SplunkTrust
SplunkTrust

Oh then mostly the problem is of connectivity do check telnet test.
Also Check for forwarder logs on second server
Are they populating?

0 Karma
Highlighted

Re: Splunk Forwarding doesn't forward data

Path Finder

thanks mayurr98. telnet is connecting.

We are able to forward data to server 2 from

  1. another forwarder on another machine
  2. server 1 => forwarding the information

that is what surprises us.

We'll check the server side logs reg. forwarder.

0 Karma
Highlighted

Re: Splunk Forwarding doesn't forward data

Path Finder

Hi Mayurr98, the tcp connection from server2 to destination splunk server on ports 8089 and 9997 are working as expected.

0 Karma
Highlighted

Re: Splunk Forwarding doesn't forward data

SplunkTrust
SplunkTrust

hey I faced the same problem while getting data in from on TCP
Everything was working fine. So the problem got solved by enabling IP forwarding on the server.
Refer this link, and let me know:
http://www.ducea.com/2006/08/01/how-to-enable-ip-forwarding-in-linux/

0 Karma
Highlighted

Re: Splunk Forwarding doesn't forward data

Communicator

You can forward only to one destination that way, if you'd like to forward the data to two indexers, then you have to combine it.

 [tcpout]
 defaultGroup=awsprod,awsdev

 [tcpout:awsprod]
 server=<server1-ip>:9997,server=<server2-ip>:9997
 useACK = true

I assume you don't have Index replication enabled.

Highlighted

Re: Splunk Forwarding doesn't forward data

Path Finder

From the docs, if we give server list in comma separated fashion, the data will be load balanced between two receivers. Please confirm if my understanding is correct.

# Specify a target group made up of two receivers.  In this case, the data will
# be distributed using AutoLB between these two receivers.  You can specify as
# many receivers as you wish here. You can combine host name and IP if you
# wish.
# NOTE: Do not use this configuration with SplunkLightForwarder.

[tcpout:group3]
server=myhost.Splunk.com:9997,10.1.1.197:6666

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Outputsconf

0 Karma
Highlighted

Re: Splunk Forwarding doesn't forward data

Path Finder

Sorry if we have not added sufficient details earlier. We wish to clone data to both the servers as they are QA and Prod respectively.

0 Karma