Getting Data In

Forwarder not sending data to indexer

OldManEd
Builder

I had a little test environment set up to test forwarding to a test indexer and it worked fine. Now, I altered the files to send data to our production indexers, and although the forwarder appears to be connecting to the indexers, I'm seeing no data. I'm wondering if I need to alter the current forwarder outputs.conf file to include a second [tcpoout] stanza like below to get this to work;

[tcpout]
defaultGroup = ProdIndexerList

[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997

Below are the specifics;

Current messages from the forwarder splunkd.log file where it looks like the forwarder is connecting to the suite of indexers successfully:

07-17-2014 17:46:10.582 +0000 INFO  ThruputProcessor - Current data throughput (276 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
07-17-2014 17:46:58.901 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xx.01:9997
07-17-2014 17:47:44.737 +0000 INFO  BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.2'.
07-17-2014 17:48:59.002 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.02:9997
07-17-2014 17:49:22.093 +0000 INFO  BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.1'.
07-17-2014 17:50:59.423 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.03:9997
07-17-2014 17:52:59.355 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.04:9997

==============================================================================================

Current forwarder inputs.conf file;

[default]
host = forwarder_host_name

[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>

Current forwarder outputs.conf file;

[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997

Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/indexes.conf file;

[ABClogs]
disabled=false
homePath = $SPLUNK_DB/ABCdb/db
coldPath = $SPLUNK_DB/ABCdb/colddb
thawedPath = $SPLUNK_DB/ABCdb/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 100000

Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/props.conf file;

[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1

After all the conf files were update, all the indexers and forwarders were restarted but not the search heads.

=====================================================================================

Below is the configuration I had on the test environment that did work:

Original test forwarder inputs.conf file;

[default]
host = forwarder_host_name

[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>

Original test forwarder outputs.conf file;

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
Server=xx.xx.x.99:9997

Original test indexer indexes.conf file;

[ABClogs]
coldPath = $SPLUNK_DB/ABCdb/colddb
homePath = $SPLUNK_DB/ABCdb/db
maxTotalDataSizeMB = 5000
thawedPath = $SPLUNK_DB/ABCdb/thaweddb

Original test indexer props.conf file;

[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1
0 Karma
1 Solution

OldManEd
Builder

The suggestion by Strive to add back the stanza below worked;

[tcpout]
defaultGroup = ProdIndexerList

BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.

View solution in original post

OldManEd
Builder

The suggestion by Strive to add back the stanza below worked;

[tcpout]
defaultGroup = ProdIndexerList

BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.

OldManEd
Builder

Strive,
That did it. Thanks a lot.

0 Karma

strive
Influencer

Yes you need to add the second tcpout stanza to your outputs.conf file, similar to your test environment.
Splunk documentation states that "Starting with 4.2, the [tcpout] stanza is no longer required." But it did not work for me. It worked when i had both the stanzas in my outputs.conf file

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...