I had a little test environment set up to test forwarding to a test indexer and it worked fine. Now, I altered the files to send data to our production indexers, and although the forwarder appears to be connecting to the indexers, I'm seeing no data. I'm wondering if I need to alter the current forwarder outputs.conf file to include a second [tcpoout] stanza like below to get this to work;
[tcpout]
defaultGroup = ProdIndexerList
[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997
Below are the specifics;
Current messages from the forwarder splunkd.log file where it looks like the forwarder is connecting to the suite of indexers successfully:
07-17-2014 17:46:10.582 +0000 INFO ThruputProcessor - Current data throughput (276 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
07-17-2014 17:46:58.901 +0000 INFO TcpOutputProc - Connected to idx=xx.xxx.xx.01:9997
07-17-2014 17:47:44.737 +0000 INFO BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.2'.
07-17-2014 17:48:59.002 +0000 INFO TcpOutputProc - Connected to idx=xx.xxx.xxx.02:9997
07-17-2014 17:49:22.093 +0000 INFO BatchReader - Removed from queue file='/opt/splunkforwarder/var/log/splunk/metrics.log.1'.
07-17-2014 17:50:59.423 +0000 INFO TcpOutputProc - Connected to idx=xx.xxx.xxx.03:9997
07-17-2014 17:52:59.355 +0000 INFO TcpOutputProc - Connected to idx=xx.xxx.xxx.04:9997
==============================================================================================
Current forwarder inputs.conf file;
[default]
host = forwarder_host_name
[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>
Current forwarder outputs.conf file;
[tcpout:ProdIndexerList]
autoLB=true
autoLBFrequency=120
server=xx.xxx.xxx.01:9997, xx.xxx.xxx.02:9997, xx.xxx.xxx.03:9997, xx.xxx.xxx.04:9997
Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/indexes.conf file;
[ABClogs]
disabled=false
homePath = $SPLUNK_DB/ABCdb/db
coldPath = $SPLUNK_DB/ABCdb/colddb
thawedPath = $SPLUNK_DB/ABCdb/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 100000
Current indexer(s) /opt/splunk/etc/apps/cricketIndexers/local/props.conf file;
[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1
After all the conf files were update, all the indexers and forwarders were restarted but not the search heads.
=====================================================================================
Below is the configuration I had on the test environment that did work:
Original test forwarder inputs.conf file;
[default]
host = forwarder_host_name
[monitor:///data_directory/ABC_*File.log.csv]
index=ABClogs
sourcetype=ABCtype
ignoreOlderThan = 2d
crcSalt=<SOURCE>
Original test forwarder outputs.conf file;
[tcpout]
defaultGroup = splunkssl
[tcpout:splunkssl]
Server=xx.xx.x.99:9997
Original test indexer indexes.conf file;
[ABClogs]
coldPath = $SPLUNK_DB/ABCdb/colddb
homePath = $SPLUNK_DB/ABCdb/db
maxTotalDataSizeMB = 5000
thawedPath = $SPLUNK_DB/ABCdb/thaweddb
Original test indexer props.conf file;
[ABCtype]
CHECK_FOR_HEADER = true
HEADER_MODE = firstline
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TZ=UTC
pulldown_type = 1
The suggestion by Strive to add back the stanza below worked;
[tcpout]
defaultGroup = ProdIndexerList
BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.
The suggestion by Strive to add back the stanza below worked;
[tcpout]
defaultGroup = ProdIndexerList
BTW, we have Splunk 5.0.5 installed. The documentation that suggests that the [tcpout] stanza is no longer required is not 100% correct.
Strive,
That did it. Thanks a lot.
Yes you need to add the second tcpout stanza to your outputs.conf file, similar to your test environment.
Splunk documentation states that "Starting with 4.2, the [tcpout] stanza is no longer required." But it did not work for me. It worked when i had both the stanzas in my outputs.conf file