Getting Data In

Forwarder not sending data to indexer

OldManEd
Builder

I had a little test environment set up to test forwarding to a test indexer and it worked fine. Now, I altered the files to send data to our production indexers, and although the forwarder appears to be connecting to the indexers, I'm seeing no data. I'm wondering if I need to alter the current forwarder outputs.conf file to include a second [tcpoout] stanza like below to get this to work;

[tcpout]
defaultGroup = ProdIndexerList

[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997

Below are the specifics;

Current messages from the forwarder splunkd.log file where it looks like the forwarder is connecting to the suite of indexers successfully:

07-17-2014 17:46:10.582 +0000 INFO  ThruputProcessor - Current data throughput (276 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
07-17-2014 17:46:58.901 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xx.01:9997
07-17-2014 17:47:44.737 +0000 INFO  BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.2'.
07-17-2014 17:48:59.002 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.02:9997
07-17-2014 17:49:22.093 +0000 INFO  BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.1'.
07-17-2014 17:50:59.423 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.03:9997
07-17-2014 17:52:59.355 +0000 INFO  TcpOutputProc - Connected to idx=xx.xxx.xxx.04:9997

==============================================================================================

Current forwarder inputs.conf file;

[default]
host = forwarder_host_name

[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>

Current forwarder outputs.conf file;

[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997

Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/indexes.conf file;

[ABClogs]
disabled=false
homePath = $SPLUNK_DB/ABCdb/db
coldPath = $SPLUNK_DB/ABCdb/colddb
thawedPath = $SPLUNK_DB/ABCdb/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 100000

Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/props.conf file;

[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1

After all the conf files were update, all the indexers and forwarders were restarted but not the search heads.

=====================================================================================

Below is the configuration I had on the test environment that did work:

Original test forwarder inputs.conf file;

[default]
host = forwarder_host_name

[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>

Original test forwarder outputs.conf file;

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
Server=xx.xx.x.99:9997

Original test indexer indexes.conf file;

[ABClogs]
coldPath = $SPLUNK_DB/ABCdb/colddb
homePath = $SPLUNK_DB/ABCdb/db
maxTotalDataSizeMB = 5000
thawedPath = $SPLUNK_DB/ABCdb/thaweddb

Original test indexer props.conf file;

[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1
0 Karma
1 Solution

OldManEd
Builder

The suggestion by Strive to add back the stanza below worked;

[tcpout]
defaultGroup = ProdIndexerList

BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.

View solution in original post

OldManEd
Builder

The suggestion by Strive to add back the stanza below worked;

[tcpout]
defaultGroup = ProdIndexerList

BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.

OldManEd
Builder

Strive,
That did it. Thanks a lot.

0 Karma

strive
Influencer

Yes you need to add the second tcpout stanza to your outputs.conf file, similar to your test environment.
Splunk documentation states that "Starting with 4.2, the [tcpout] stanza is no longer required." But it did not work for me. It worked when i had both the stanzas in my outputs.conf file

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...