topic Re: Splunk Forwarding doesn't forward data in Getting Data In

Splunk Forwarding doesn't forward data

ykpramodhcbt — Sat, 16 Dec 2017 12:42:01 GMT

We have a single data source from which we want to forward clone data to - splunk server 1(prod) and splunk server 2(qa).

The data seems to go to splunk server 1 fine but doesn't get forwarded to splunk server 2. We don't anything wrong in the log file too.

splunk list forward-server lists both the servers

outputs.conf (Windows Forwarder)

[tcpout]
defaultGroup=awsprod,awsdev

[tcpout:awsprod]
server=<server1-ip>:9997
useACK = true

[tcpout:awsdev]
server=<server2-ip>:9997
useACK = true

As a work around, we have put a forward stanza on splunk server 1(prod) to forward data to splunk server 2(qa) and it seems to work fine.
When we try to forward data from other machines to server2 (qa), it seems to work fine.

Any suggestions are highly appreciated.

PS: More details on cloning and server details - qa/prod added.

Re: Splunk Forwarding doesn't forward data

mayurr98 — Sat, 16 Dec 2017 13:34:39 GMT

There is 99% chance you might have misconfigured forwarder.
on indexers search app look for the output of below query

index=_internal host=forwarder

If you get the data it means you have configure the forwarder properly. If you get the logs then look for errors in those logs.
Refer this link:
http://docs.splunk.com/Documentation/Forwarder/7.0.1/Forwarder/Configuretheuniversalforwarder

Also check the output at the forwarder cli in order to check the connectivity

telnet indexer-ip 8089
telnet indexer-ip 9997

Check if you have enabled forwarder receiving port 9997 on both indexers.
Also check if the monitor stanza that you have written is correct or not!
Let me know if this helps!!

Re: Splunk Forwarding doesn't forward data

ykpramodhcbt — Sat, 16 Dec 2017 13:54:04 GMT

Hi mayurr98,

Thanks for the note.

Here is the inputs.conf

[monitor://d:\Carbynetech.csv]
disabled=false
index=indexname

What surprises us is that data is getting forwarded to one server. We will do telnet test and report our findings.

regards
Pramodh

Re: Splunk Forwarding doesn't forward data

mayurr98 — Sat, 16 Dec 2017 14:08:15 GMT

Oh then mostly the problem is of connectivity do check telnet test.
Also Check for forwarder logs on second server
Are they populating?

Re: Splunk Forwarding doesn't forward data

ykpramodhcbt — Sat, 16 Dec 2017 19:00:57 GMT

thanks mayurr98. telnet is connecting.

We are able to forward data to server 2 from

another forwarder on another machine
server 1 => forwarding the information

that is what surprises us.

We'll check the server side logs reg. forwarder.

Re: Splunk Forwarding doesn't forward data

ykpramodhcbt — Wed, 20 Dec 2017 13:28:36 GMT

Hi Mayurr98, the tcp connection from server2 to destination splunk server on ports 8089 and 9997 are working as expected.

Re: Splunk Forwarding doesn't forward data

Elsurion — Wed, 20 Dec 2017 13:55:56 GMT

You can forward only to one destination that way, if you'd like to forward the data to two indexers, then you have to combine it.

 [tcpout]
 defaultGroup=awsprod,awsdev

 [tcpout:awsprod]
 server=<server1-ip>:9997,server=<server2-ip>:9997
 useACK = true

I assume you don't have Index replication enabled.

Re: Splunk Forwarding doesn't forward data

ykpramodhcbt — Thu, 21 Dec 2017 04:39:19 GMT

From the docs, if we give server list in comma separated fashion, the data will be load balanced between two receivers. Please confirm if my understanding is correct.

# Specify a target group made up of two receivers.  In this case, the data will
# be distributed using AutoLB between these two receivers.  You can specify as
# many receivers as you wish here. You can combine host name and IP if you
# wish.
# NOTE: Do not use this configuration with SplunkLightForwarder.

[tcpout:group3]
server=myhost.Splunk.com:9997,10.1.1.197:6666

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Outputsconf

Re: Splunk Forwarding doesn't forward data

harsmarvania57 — Thu, 21 Dec 2017 05:01:25 GMT

Based on configuration which you have provided, this will clone data to both the Indexer (Server -1 and Server - 2). What you want to achieve, do you want to send data to both the indexer in load balance way (Not cloning of data) then answer provided by @Elsurion is correct with minor modification.

[tcpout]
defaultGroup=awsprod

[tcpout:awsprod]
server=<server1-ip>:9997,<server2-ip>:9997
useACK = true

Re: Splunk Forwarding doesn't forward data

ykpramodhcbt — Thu, 21 Dec 2017 05:38:55 GMT

Our requirement is to clone data to both the servers. The servers are QA and Prod instances respectively.

The surprising part is data is not reaching QA and as a work around we have setup forwarding from Prod to QA.

Re: Splunk Forwarding doesn't forward data

ykpramodhcbt — Thu, 21 Dec 2017 05:41:29 GMT

Sorry if we have not added sufficient details earlier. We wish to clone data to both the servers as they are QA and Prod respectively.

Re: Splunk Forwarding doesn't forward data

harsmarvania57 — Thu, 21 Dec 2017 05:42:00 GMT

In that case configuration which you have provided is correct and I am assuming you are not using _TCP_ROUTING in your monitor stanza in inputs.conf

Can you please check from your UF to Server -2 network connectivity using telnet command telnet Server_2_IP 9997 ?

Re: Splunk Forwarding doesn't forward data

ykpramodhcbt — Thu, 21 Dec 2017 05:44:29 GMT

Thanks for your note.

We are not using _TCP_ROUTING

telnet Server2 9997 is working fine.

Re: Splunk Forwarding doesn't forward data

nikita_p — Thu, 21 Dec 2017 05:58:37 GMT

Yes, forwarder will send data in a load balanced way and data will not be cloned if you are using below configuration:-
[tcpout]
defaultGroup=awsprod,awsdev

[tcpout:awsprod]
server=:9997,server=:9997
useACK = true

But if you want to clone could you try the below configuration in your outputs.conf
[tcpout]
defaultGroup = awsprod

[tcpout-server://:9997]

[tcpout-server://1:9997]

Re: Splunk Forwarding doesn't forward data

harsmarvania57 — Thu, 21 Dec 2017 06:10:55 GMT

Ok, can you please try to run below query on Server-2 (Indexer-2) so that we can check whether you are receiving data on Server-2 from UF or not

index=_internal host=Sever2 source=*metrics.log* group=per_host_thruput series=UF_FQDN

Re: Splunk Forwarding doesn't forward data

mayurr98 — Thu, 21 Dec 2017 06:40:20 GMT

hey I faced the same problem while getting data in from on TCP
Everything was working fine. So the problem got solved by enabling IP forwarding on the server.
Refer this link, and let me know:
http://www.ducea.com/2006/08/01/how-to-enable-ip-forwarding-in-linux/

Re: Splunk Forwarding doesn't forward data

nikita_p — Thu, 21 Dec 2017 06:49:56 GMT

Hi,
Sorry there was some typo in my outputs.conf
[tcpout]
defaultGroup = awsprod

[tcpout-server://server1-ip:9997]
[tcpout-server://server2-ip:9997]

Re: Splunk Forwarding doesn't forward data

ykpramodhcbt — Thu, 21 Dec 2017 07:07:48 GMT

We will try this and we will update you.