Forwarder not sending data to indexer

OldManEd — Thu, 17 Jul 2014 19:08:18 GMT

I had a little test environment set up to test forwarding to a test indexer and it worked fine. Now, I altered the files to send data to our production indexers, and although the forwarder appears to be connecting to the indexers, I'm seeing no data. I'm wondering if I need to alter the current forwarder outputs.conf file to include a second [tcpoout] stanza like below to get this to work;

[tcpout]
defaultGroup = ProdIndexerList

[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997

Below are the specifics;

Current messages from the forwarder splunkd.log file where it looks like the forwarder is connecting to the suite of indexers successfully:

07-17-2014 17:46:10.582 +0000 INFO  ThruputProcessor - Current data throughput (276 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
07-17-2014 17:46:58.901 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xx.01:9997
07-17-2014 17:47:44.737 +0000 INFO  BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.2'.
07-17-2014 17:48:59.002 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.02:9997
07-17-2014 17:49:22.093 +0000 INFO  BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.1'.
07-17-2014 17:50:59.423 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.03:9997
07-17-2014 17:52:59.355 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.04:9997

==============================================================================================

Current forwarder inputs.conf file;

[default]
host = forwarder_host_name

[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>

Current forwarder outputs.conf file;

[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997

Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/indexes.conf file;

[ABClogs]
disabled=false
homePath = $SPLUNK_DB/ABCdb/db
coldPath = $SPLUNK_DB/ABCdb/colddb
thawedPath = $SPLUNK_DB/ABCdb/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 100000

Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/props.conf file;

[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1

After all the conf files were update, all the indexers and forwarders were restarted but not the search heads.

=====================================================================================

Below is the configuration I had on the test environment that did work:

Original test forwarder inputs.conf file;

[default]
host = forwarder_host_name

[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>

Original test forwarder outputs.conf file;

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
Server=xx.xx.x.99:9997

Original test indexer indexes.conf file;

[ABClogs]
coldPath = $SPLUNK_DB/ABCdb/colddb
homePath = $SPLUNK_DB/ABCdb/db
maxTotalDataSizeMB = 5000
thawedPath = $SPLUNK_DB/ABCdb/thaweddb

Original test indexer props.conf file;

[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1

Re: Forwarder not sending data to indexer

strive — Fri, 18 Jul 2014 04:11:45 GMT

Yes you need to add the second tcpout stanza to your outputs.conf file, similar to your test environment.
Splunk documentation states that "Starting with 4.2, the [tcpout] stanza is no longer required." But it did not work for me. It worked when i had both the stanzas in my outputs.conf file

Re: Forwarder not sending data to indexer

OldManEd — Fri, 18 Jul 2014 19:17:12 GMT

Strive,
That did it. Thanks a lot.

Re: Forwarder not sending data to indexer

OldManEd — Fri, 18 Jul 2014 19:20:49 GMT

The suggestion by Strive to add back the stanza below worked;

[tcpout]
defaultGroup = ProdIndexerList

BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.

topic Re: Forwarder not sending data to indexer in Getting Data In

Forwarder not sending data to indexer

Re: Forwarder not sending data to indexer

Re: Forwarder not sending data to indexer

Re: Forwarder not sending data to indexer