This was working originally, however, I realised I had put in the wrong sourcetype, I changed the inputs.conf files to remove the sourcetype, cleaned the index, and then reloaded the deployment server, forwarder and indexers to re-index the data with the default sourcetype...
Now the forwarders have grabbed the right inputs.conf and outputs.conf files, and the logs indicate the are connecting nicely to the indexers. Unfortunately, the indexer logs are still empty 😞 Thought it might be an issue with the index so I tried creating a new 1, web1 but its still not working 😞 and I see absolutly nothing in the indexer logs to indicate the forwarders traffic is being received, in addition to this there are some strange "failed to build" in the indexer logs, is there anything else I can check?
Please see my config and logs below:
FORWARDER1 INPUTS.CONF
ubuntu@ip-10-171-2-174:/$ sudo cat /opt/splunkforwarder/etc/apps/webservers4/local/inputs.conf
[monitor:///opt/log/www1]
index = web1
host = www1
[monitor:///opt/log/www2]
index = web1
host = www2
[monitor:///opt/log/www3]
index = web1
host = www3
FORWARDER1 OUTPUTS.CONF
ubuntu@ip-10-171-2-174:/$ sudo cat /opt/splunkforwarder/etc/apps/webservers4/local/outputs.conf
[tcpout]
defaultGroup=splunk
[tcpout:splunk]
server= :9997, :9997
FORWARDER1 LOGS:
ubuntu@ip-10-171-2-174:/$ sudo tac /opt/splunkforwarder/var/log/splunk/splunkd.l og | more
09-03-2012 11:46:14.503 +0000 INFO TcpOutputProc - Connected to idx=10.171.82.8
8:9997
09-03-2012 11:46:14.398 +0000 INFO TcpOutputProc - Connected to idx=10.166.203.
114:9997
09-03-2012 11:45:14.441 +0000 INFO TcpOutputProc - Connected to idx=10.171.82.8
8:9997
09-03-2012 11:45:14.395 +0000 INFO TcpOutputProc - Connected to idx=10.166.203.
114:9997
09-03-2012 11:44:15.334 +0000 INFO TcpOutputProc - Connected to idx=10.171.82.8
8:9997
09-03-2012 11:44:14.443 +0000 INFO TcpOutputProc - Connected to idx=10.166.203.
114:9997
09-03-2012 11:43:16.924 +0000 INFO TcpOutputProc - Connected to idx=10.171.82.8
8:9997
09-03-2012 11:42:45.020 +0000 INFO TcpOutputProc - Connected to idx=10.166.203.
114:9997
09-03-2012 11:42:44.593 +0000 INFO TcpOutputProc - Connected to idx=10.171.82.8
8:9997
09-03-2012 11:41:44.958 +0000 INFO TcpOutputProc - Connected to idx=10.166.203.
114:9997
09-03-2012 11:41:44.589 +0000 INFO TcpOutputProc - Connected to idx=10.171.82.8
8:9997
09-03-2012 11:40:44.898 +0000 INFO TcpOutputProc - Connected to idx=10.166.203.
INDEXER1 LOGS:
ubuntu@ip-10-166-203-114:~$ sudo tac /opt/splunk/var/log/splunk/splunkd.log | more
09-03-2012 11:31:35.464 +0000 INFO IndexProcessor - reloading index config: end
09-03-2012 11:31:35.464 +0000 INFO IndexProcessor - request state change RECONFIGURING to RUN
09-03-2012 11:31:35.464 +0000 INFO databasePartitionPolicy - Bucket /opt/splunk/var/lib/splunk/audit/db/db_1341834080_1341832188_0 has events at least 55 days old, past max 30 days, will not repair --bloom-only
09-03-2012 11:31:35.463 +0000 INFO databasePartitionPolicy - currentId for /opt/splunk/var/lib/splunk/proxy/db after openDatabases = 0; 0 buckets being rebuilt
09-03-2012 11:31:35.463 +0000 INFO databasePartitionPolicy - rebuildMetadata called: full=true path=/opt/splunk/var/lib/splunk/proxy/db reason=initopenMetaData failed
09-03-2012 11:31:35.463 +0000 WARN databasePartitionPolicy - failed to open metadata for /opt/splunk/var/lib/splunk/proxy/db, will attempt full rebuild
09-03-2012 11:31:35.463 +0000 INFO databasePartitionPolicy - CREATION TIME for /opt/splunk/var/lib/splunk/proxy/db : 1346671895
09-03-2012 11:31:35.463 +0000 INFO databasePartitionPolicy - index proxy initialized with [300,60,188697600,,/opt/frozen,,2097152000,20,true,20000,5,5,false,3,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,
18446744073709551615ms,2592000,true,false]
09-03-2012 11:31:35.463 +0000 INFO HotDBManager - index=proxy Setting hot mgr params: maxHotSpanSecs=7776000 snapBucketTimespans=false maxHotBuckets=3 maxDataSizeBytes=2097152000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
09-03-2012 11:31:35.463 +0000 INFO IndexProcessor - indexes.conf - indexThreads param autotuned to 2
09-03-2012 11:31:35.463 +0000 INFO IndexProcessor - initializing with fullInit=1, reloading=1
09-03-2012 11:31:35.463 +0000 INFO IndexProcessor - Reloading index config: shutdown subordinate threads, now restarting
09-03-2012 11:31:35.462 +0000 INFO IndexProcessor - Got a list of 1 added, modified, or removed indexes
09-03-2012 11:31:35.462 +0000 INFO IndexProcessor - setting process pool max groups low priority=1
09-03-2012 11:31:35.462 +0000 INFO IndexProcessor - setting process pool max groups=20
09-03-2012 11:31:35.461 +0000 INFO IndexProcessor - request state change RUN to RECONFIGURING
09-03-2012 11:31:35.461 +0000 INFO IndexProcessor - reloading index config: start
09-03-2012 11:31:35.459 +0000 INFO IndexProcessor - reloading index config: request received
09-03-2012 11:27:18.044 +0000 INFO IndexProcessor - reloading index config: end
09-03-2012 11:27:18.044 +0000 INFO IndexProcessor - request state change RECONFIGURING to RUN
09-03-2012 11:27:18.044 +0000 INFO databasePartitionPolicy - Bucket /opt/splunk/var/lib/splunk/audit/db/db_1341834080_1341832188_0 has events at least 55 days old, past max 30 days, will not repair --bloom-only
09-03-2012 11:27:18.043 +0000 INFO databasePartitionPolicy - currentId for /opt/splunk/var/lib/splunk/web1/db after openDatabases = 0; 0 buckets being rebuilt
09-03-2012 11:27:18.043 +0000 INFO databasePartitionPolicy - rebuildMetadata called: full=true path=/opt/splunk/var/lib/splunk/web1/db reason=initopenMetaData failed
09-03-2012 11:27:18.043 +0000 WARN databasePartitionPolicy - failed to open metadata for /opt/splunk/var/lib/splunk/web1/db, will attempt full rebuild
09-03-2012 11:27:18.043 +0000 INFO databasePartitionPolicy - CREATION TIME for /opt/splunk/var/lib/splunk/web1/db : 1346671638
09-03-2012 11:27:18.042 +0000 INFO databasePartitionPolicy - index web1 initialized with [300,60,188697600,,/opt/frozen,,2097152000,20,true,30000,5,5,false,3,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,1
8446744073709551615ms,2592000,true,false]
09-03-2012 11:27:18.042 +0000 INFO HotDBManager - index=web1 Setting hot mgr params: maxHotSpanSecs=7776000 snapBucketTimespans=false maxHotBuckets=3 maxDataSizeBytes=2097152000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
09-03-2012 11:27:18.042 +0000 INFO IndexProcessor - indexes.conf - indexThreads param autotuned to 2
09-03-2012 11:27:18.042 +0000 INFO IndexProcessor - initializing with fullInit=1, reloading=1
09-03-2012 11:27:18.042 +0000 INFO IndexProcessor - Reloading index config: shutdown subordinate threads, now restarting
09-03-2012 11:27:18.042 +0000 INFO IndexProcessor - Got a list of 1 added, modified, or removed indexes
09-03-2012 11:27:18.042 +0000 INFO IndexProcessor - setting process pool max groups low priority=1
09-03-2012 11:27:18.042 +0000 INFO IndexProcessor - setting process pool max groups=20
09-03-2012 11:27:18.041 +0000 INFO IndexProcessor - request state change RUN to RECONFIGURING
09-03-2012 11:27:18.041 +0000 INFO IndexProcessor - reloading index config: start
09-03-2012 11:27:18.039 +0000 INFO IndexProcessor - reloading index config: request received
09-03-2012 11:02:44.568 +0000 INFO databasePartitionPolicy - Completed regenerating the bucket manifest.
... View more