I have the following in my outputs.conf. I took this config from another working heavy forwarder, and this is from new heavy forwarder I set it up recently.
[tcpout]
connectionTimeout = 45
defaultGroup = all_indexers
indexAndForward = 0
forwardedindex.0.whitelist = .*
[tcpout:all_indexers]
autoLB = true
server = heavyforwarder.hostname:9997
[tcpout-server://heavyforwarder.hostname:9997]
I believe this is correct as per your recommendation. Is it something I'm missing. Here is errors from end client machine
04-08-2015 15:53:57.351 -0700 INFO IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=true fishbucket=true partitions=true
04-08-2015 15:53:57.351 -0700 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
04-08-2015 15:53:57.357 -0700 INFO HttpPubSubConnection - SSL connection with id: connection_172.29.26.66_8089_n01scl001.aap.csaa.pri_n01scl001_77D5E0FB-3193-4598-9A96-2449BB33930A
04-08-2015 15:53:57.371 -0700 WARN DistributedPeerManager - feature=DistSearch not enabled for your license level
04-08-2015 15:53:57.371 -0700 INFO IndexProcessor - running splunkd specific init
04-08-2015 15:53:57.371 -0700 INFO loader - Initializing from configuration
04-08-2015 15:53:57.372 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a8282e180
04-08-2015 15:53:57.385 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a8202e180
04-08-2015 15:53:57.385 -0700 INFO TcpOutputProc - Initializing with fwdtype=lwf
04-08-2015 15:53:57.385 -0700 INFO ServerRoles - Declared role=lightweight_forwarder.
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to n01sml105.aap.csaa.pri:9997
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - tcpout group all_indexers using Auto load balanced forwarding
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - Group all_indexers initialized with maxQueueSize=512000 in bytes.
04-08-2015 15:53:57.403 -0700 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to n01sml105.aap.csaa.pri:9997
04-08-2015 15:53:57.403 -0700 INFO TcpOutputProc - tcpout group heavy_forwarders using Auto load balanced forwarding
04-08-2015 15:53:57.403 -0700 INFO TcpOutputProc - Group heavy_forwarders initialized with maxQueueSize=512000 in bytes.
04-08-2015 15:53:57.403 -0700 ERROR TcpOutputProc - the 'defaultGroup' property contains an invalid group name - heavy_forwarder - skipping
04-08-2015 15:53:57.403 -0700 WARN TcpOutputProc - Default groupList has not yet been calculated!
04-08-2015 15:53:57.403 -0700 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
04-08-2015 15:53:57.403 -0700 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
04-08-2015 15:53:57.403 -0700 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
04-08-2015 15:53:57.430 -0700 WARN HttpPubSubConnection - Received message for an unsubscribed channel: deploymentServer/phoneHome/default/reply/n01scl001/77D5E0FB-3193-4598-9A96-2449BB33930A
04-08-2015 15:53:57.440 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a81045180
04-08-2015 15:53:57.440 -0700 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
04-08-2015 15:53:57.441 -0700 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
04-08-2015 15:53:57.441 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a88c5fc80
04-08-2015 15:53:57.491 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a87c56180
04-08-2015 15:53:57.491 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a87c56480
04-08-2015 15:53:57.551 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a87c56600
04-08-2015 15:53:57.551 -0700 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file
04-08-2015 15:53:57.551 -0700 INFO PipelineComponent - Launching the pipelines.
04-08-2015 15:53:57.552 -0700 INFO loader - Limiting REST HTTP server to 5461 sockets
04-08-2015 15:53:57.553 -0700 INFO loader - Limiting REST HTTP server to 1299 threads
04-08-2015 15:53:57.597 -0700 INFO TailingProcessor - TailWatcher initializing...
04-08-2015 15:53:57.597 -0700 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor:///opt/elasticsearch/apache-tomcat-7.0.53/logs/catalina.out.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Adding watch on path: /opt/elasticsearch/apache-tomcat-7.0.53/logs/catalina.out.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/etc/splunk.version.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/var/log/splunk.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/var/spool/splunk.
04-08-2015 15:53:57.598 -0700 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
04-08-2015 15:53:57.608 -0700 INFO TcpOutputProc - Connected to idx=172.29.28.72:9997
04-08-2015 15:53:57.608 -0700 INFO TcpOutputProc - Connected to idx=172.29.28.72:9997
04-08-2015 15:53:57.673 -0700 INFO WatchedFile - Will begin reading at offset=16460 for file='/opt/splunk/splunkforwarder/var/log/splunk/splunkd-utility.log'.
04-08-2015 15:53:57.700 -0700 INFO WatchedFile - Will begin reading at offset=62969 for file='/opt/splunk/splunkforwarder/var/log/splunk/audit.log'.
04-08-2015 15:53:57.703 -0700 INFO WatchedFile - Will begin reading at offset=1317 for file='/opt/splunk/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
04-08-2015 15:53:57.775 -0700 INFO WatchedFile - Will begin reading at offset=19758958 for file='/opt/splunk/splunkforwarder/var/log/splunk/metrics.log'.
04-08-2015 15:54:33.398 -0700 WARN DC:PhonehomeThread - No response to handshake for too long; starting over.
04-08-2015 15:54:33.398 -0700 WARN DC:PhonehomeThread - No response to handshake for too long; starting over.
04-08-2015 15:54:33.445 -0700 INFO DC:HandshakeReplyHandler - Handshake done.
04-08-2015 15:55:33.491 -0700 INFO DC:HandshakeReplyHandler - Handshake done.
... View more