I am new to Splunk and trying to troubleshoot the "splunk newbies" dreaded "Search peer 'xxx' has the following message: Too many streaming errors to target='target':9997. Not rolling hot buckets on further errors to this target." issue on a distributed deployment built as follows: Search layer composed of a search head cluster with 3 search heads and the cluster's deployer; Index layer composed of an indexer cluster with 5 indexers the cluster's master node - cluster is configured using replication factor = 5 Forwarder layer currently deployed with 3 forwarders with a deployment server (not yet doing anything really with these forwarders until I get the issues identified in this post resolved. I have sos app v 3.2.1 deployed on the shc members (applied successfully using cluster-bundle from the shc deployer) and configured to enable the 2 sos input scripts (sof_sos.sh and ps_sos.sh) on each of the 3 shc members. BTW, after doing this I realized I could and probably will make changes to the /apps/sos/local/inputs.conf on the shc deployer for better, more time efficient configuration management. I also deployed sideview utils v 3.3.2 to the shc members using cluster-bundle, although truth be told I inadvertently deployed sos apps BEFORE sideview utils, although I didn't configure sos before correcting this deployment fopaux (or should we call that foobar - lol). I also deployed TA-sos v 2.0.5 on the 5 idx-cluster peers using the idx-cluster master node (apply cluster-bundle) and configured each idx-cluster peer to enable the 2 sos input scripts (sof_sos.sh and ps_sos.sh). On 1 idx-cluster peer I have the following sets of configurations dumped using btool: From indexes - (only showing configs that are not from the default indexes.conf) /opt/splunk/etc/slave-apps/_cluster/default/indexes.conf [_audit] stanza /opt/splunk/etc/slave-apps/_cluster/default/indexes.conf repFactor = auto /opt/splunk/etc/slave-apps/_cluster/default/indexes.conf [_internal] /opt/splunk/etc/slave-apps/_cluster/default/indexes.conf repFactor = auto /opt/splunk/etc/system/default/indexes.conf [default] /opt/splunk/etc/system/default/indexes.conf repFactor = 0 /opt/splunk/etc/slave-apps/_cluster/default/indexes.conf [main] /opt/splunk/etc/slave-apps/_cluster/default/indexes.conf repFactor = auto /opt/splunk/etc/slave-apps/TA-sos/local/indexes.conf [sos] /opt/splunk/etc/slave-apps/TA-sos/local/indexes.conf coldPath = $SPLUNK_DB/sos/colddb /opt/splunk/etc/system/default/indexes.conf defaultDatabase = main /opt/splunk/etc/slave-apps/TA-sos/local/indexes.conf disabled = 0 /opt/splunk/etc/slave-apps/TA-sos/local/indexes.conf frozenTimePeriodInSecs = 2419200 /opt/splunk/etc/slave-apps/TA-sos/local/indexes.conf homePath = $SPLUNK_DB/sos/db /opt/splunk/etc/slave-apps/TA-sos/local/indexes.conf repFactor = auto From idx cluster peer's inputs: /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf [script:///opt/splunk/etc/slave-apps/TA-sos/bin/lsof_sos.sh] /opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864 /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf disabled = 0 /opt/splunk/etc/slave-apps/_cluster/local/inputs.conf host = ip-172-31-26-237 /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf index = sos /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf interval = 600 /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf source = lsof_sos /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf sourcetype = lsof /opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864 /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf disabled = 0 /opt/splunk/etc/slave-apps/_cluster/local/inputs.conf host = ip-172-31-26-237 /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf index = sos /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf interval = 5 /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf source = ps_sos /opt/splunk/etc/slave-apps/TA-sos/local/inputs.conf sourcetype = ps /opt/splunk/etc/system/default/inputs.conf [splunktcp] /opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864 /opt/splunk/etc/system/default/inputs.conf acceptFrom = * /opt/splunk/etc/system/default/inputs.conf connection_host = ip /opt/splunk/etc/slave-apps/_cluster/local/inputs.conf host = ip-172-31-26-237 /opt/splunk/etc/system/default/inputs.conf index = default /opt/splunk/etc/system/default/inputs.conf route = has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue /opt/splunk/etc/slave-apps/_cluster/local/inputs.conf [splunktcp://9997] /opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864 /opt/splunk/etc/slave-apps/_cluster/local/inputs.conf disabled = 0 /opt/splunk/etc/slave-apps/_cluster/local/inputs.conf host = ip-172-31-26-237 /opt/splunk/etc/system/default/inputs.conf index = default This is the configuration for ALL the idx cluster peers. On any given idx cluster peer when I analyze the $SPLUNK_HOME/var/log/splunk/splunkd.log I get the following results (using the indicated log analysis command-line in the provided trace): root@ip-172-31-26-200:/opt/splunk/var/log/splunk# grep CMStreamingErrorJob splunkd.log* | cut -d' ' -f10,12 | sort |uniq -c | sort -nr 178 srcGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 failingGuid=27EE43F3-BD89-4BC4-9200-298F99B4275A 170 srcGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 failingGuid=5BB335C9-340F-42CD-A5C6-C8269429D10A 169 srcGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 failingGuid=93BAC151-E057-4F37-9D16-C4CAB5A971E3 163 srcGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 failingGuid=37D7692E-5D49-432E-9A6F-89C0C68FACEF 12 srcGuid=37D7692E-5D49-432E-9A6F-89C0C68FACEF failingGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 12 srcGuid=27EE43F3-BD89-4BC4-9200-298F99B4275A failingGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 11 srcGuid=5BB335C9-340F-42CD-A5C6-C8269429D10A failingGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 10 srcGuid=93BAC151-E057-4F37-9D16-C4CAB5A971E3 failingGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 2 tgtGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 failingGuid=93BAC151-E057-4F37-9D16-C4CAB5A971E3 2 tgtGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 failingGuid=5BB335C9-340F-42CD-A5C6-C8269429D10A 1 tgtGuid=93BAC151-E057-4F37-9D16-C4CAB5A971E3 failingGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 1 tgtGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 failingGuid=37D7692E-5D49-432E-9A6F-89C0C68FACEF 1 tgtGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 failingGuid=27EE43F3-BD89-4BC4-9200-298F99B4275A 1 tgtGuid=5BB335C9-340F-42CD-A5C6-C8269429D10A failingGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 1 tgtGuid=37D7692E-5D49-432E-9A6F-89C0C68FACEF failingGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 If I watch -n 1 'ls -al' the processing of hot buckets for sos, I get the following as a snap-shot in time: Every 1.0s: ls -alt Sun Apr 5 13:19:59 2015 total 132 drwx--x--x 3 splunk splunk 4096 Apr 5 13:19 . drwx------ 47 splunk splunk 4096 Apr 5 13:19 .. -rw------- 1 splunk splunk 1585 Apr 5 13:19 1428239997-1428239997-1367416106440166490.tsidx -rw------- 1 splunk splunk 1616 Apr 5 13:19 1428239992-1428239992-1367415778757555421.tsidx -rw------- 1 splunk splunk 1617 Apr 5 13:19 1428239987-1428239987-1367415450991673577.tsidx -rw------- 1 splunk splunk 1969 Apr 5 13:19 1428239982-1428239981-1367415123329070425.tsidx -rw------- 1 splunk splunk 80922 Apr 5 13:19 1428239977-1428237297-1367414798428158871.tsidx -rw------- 1 splunk splunk 291 Apr 5 13:19 Hosts.data -rw------- 1 splunk splunk 7 Apr 5 13:19 .rawSize -rw------- 1 splunk splunk 97 Apr 5 13:19 Sources.data -rw------- 1 splunk splunk 97 Apr 5 13:19 SourceTypes.data drwx------ 2 splunk splunk 4096 Apr 5 13:17 rawdata -rw------- 1 splunk splunk 41 Apr 5 12:34 Strings.data -rw------- 1 splunk splunk 67 Apr 5 12:34 bucket_info.csv I've done some minor poking about using ipTraf, iotop, netstat, tried successfully establishing tcp connections from each of the shc members to each of the idx cluster peers using netcat to port 9997. I really don't know how to go forward on this troubleshootin,g but wanted to give everything I've done so all could hopefully zero in on whatever it is I am missing here. I haven't tried yet to add additional indexers to the cluster - I was thinking about adding 3 more to make a repFactor deployment of 8 / 5 (8 peers over repFactor = 5), but think I'm going to wait on this until I get a solid operating condition on the current deployment (want to control the dependent variables in this little exercise/experiment). Thanks in advance to any and all that take the time to read my short summary of the situation - lol! ... View more

I did have a previous post - "How to get search head cluster members to forward internal data to indexer cluster? - but don't think it is working correctly - yet. I am a bit confused by one item in the "Best practice: Forward search head data to the indexer layer (http://docs.splunk.com/Documentation/Splunk/6.2.2/DistSearch/Forwardsearchheaddata) documentation step 1. which states: Make sure that all necessary indexes exist on the indexers. For example, the S.o.S app uses a scripted input that puts data into a custom index. If you install S.o.S on the search head, you need to also install the S.o.S Add-on on the indexers, to provide the indexers with the necessary index settings for the data the app generates. On the other hand, since _audit and _internal exist on indexers as well as search heads, you do not need to create separate versions of those indexes to hold the corresponding search head data. Now I do have S.O.S. configured (and running) on each of my search head cluster members, so do I also need to have S.O.S. installed on the indexers if what I want to have pushed down to the indexer layer from the search head is the _audit and _internal data? On the search head cluster member's outputs.conf (they have the same outputs.conf) I have the following in the [tcpout] maxQueueSize = auto forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_internal|_introspection) forwardedindex.filter.disable = true indexAndForward = false autoLBFrequency = 30 blockOnCloning = true compressed = false disabled = false dropClonedEventsOnQueueFull = 5 dropEventsOnQueueFull = -1 heartbeatFrequency = 30 maxFailuresPerInterval = 2 secsInFailureInterval = 1 maxConnectionsPerIndexer = 2 forceTimebasedAutoLB = false sendCookedData = true connectionTimeout = 20 readTimeout = 300 writeTimeout = 300 useACK = false blockWarnThreshold = 100 sslQuietShutdown = false defaultGroup = transtrophe_search_peers [syslog] type = udp priority = <13> dropEventsOnQueueFull = -1 maxEventSize = 1024 [indexAndForward] index = false [tcpout:transtrophe_search_peers] server=ip-172-31-20-173:9997,ip-172-31-18-186:9997,ip-172-31-22-253:9997,ip-172-31-26-200:9997,ip-172-31-20-120:9997 autoLB = true ... View more