Getting Data In

Forwarder not sending data to indexer

OldManEd
Builder

I had a little test environment set up to test forwarding to a test indexer and it worked fine. Now, I altered the files to send data to our production indexers, and although the forwarder appears to be connecting to the indexers, I'm seeing no data. I'm wondering if I need to alter the current forwarder outputs.conf file to include a second [tcpoout] stanza like below to get this to work;

[tcpout]
defaultGroup = ProdIndexerList

[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997

Below are the specifics;

Current messages from the forwarder splunkd.log file where it looks like the forwarder is connecting to the suite of indexers successfully:

07-17-2014 17:46:10.582 +0000 INFO  ThruputProcessor - Current data throughput (276 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
07-17-2014 17:46:58.901 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xx.01:9997
07-17-2014 17:47:44.737 +0000 INFO  BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.2'.
07-17-2014 17:48:59.002 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.02:9997
07-17-2014 17:49:22.093 +0000 INFO  BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.1'.
07-17-2014 17:50:59.423 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.03:9997
07-17-2014 17:52:59.355 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.04:9997

==============================================================================================

Current forwarder inputs.conf file;

[default]
host = forwarder_host_name

[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>

Current forwarder outputs.conf file;

[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997

Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/indexes.conf file;

[ABClogs]
disabled=false
homePath = $SPLUNK_DB/ABCdb/db
coldPath = $SPLUNK_DB/ABCdb/colddb
thawedPath = $SPLUNK_DB/ABCdb/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 100000

Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/props.conf file;

[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1

After all the conf files were update, all the indexers and forwarders were restarted but not the search heads.

=====================================================================================

Below is the configuration I had on the test environment that did work:

Original test forwarder inputs.conf file;

[default]
host = forwarder_host_name

[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>

Original test forwarder outputs.conf file;

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
Server=xx.xx.x.99:9997

Original test indexer indexes.conf file;

[ABClogs]
coldPath = $SPLUNK_DB/ABCdb/colddb
homePath = $SPLUNK_DB/ABCdb/db
maxTotalDataSizeMB = 5000
thawedPath = $SPLUNK_DB/ABCdb/thaweddb

Original test indexer props.conf file;

[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1
0 Karma
1 Solution

OldManEd
Builder

The suggestion by Strive to add back the stanza below worked;

[tcpout]
defaultGroup = ProdIndexerList

BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.

View solution in original post

OldManEd
Builder

The suggestion by Strive to add back the stanza below worked;

[tcpout]
defaultGroup = ProdIndexerList

BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.

OldManEd
Builder

Strive,
That did it. Thanks a lot.

0 Karma

strive
Influencer

Yes you need to add the second tcpout stanza to your outputs.conf file, similar to your test environment.
Splunk documentation states that "Starting with 4.2, the [tcpout] stanza is no longer required." But it did not work for me. It worked when i had both the stanzas in my outputs.conf file

Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...