I observe that the Rubrik Splunk Add-On v1.0.6 (January, 2020) is not listed as Compatible with Splunk 8. When can we expect that? Is version 1.0.6 Python 3 compatible now? ... View more

Upgrading from Splunk Enterprise 6.6.2 to 7.1.0 introduced two major problems: 1) Memory Usage on the Indexers would grow beyond Available Memory (and Swap space) resulting in oom-killer being invoked. oom-killer always targets the largest memory consumer, which was usually a Splunk Process. If it was a "child" process, Splunk would respawn as needed, but sometimes it would be the "parent" splunkd process, which would take down the Indexer. oom-killer would be invoked multiple times per hour, which would eventually take down an indexer multiple times each day and affected both Indexers in the Cluster. Search processes on the Indexer were dominant and far greater than any other processes. Indexing rates were low and the processing queues were almost always at 100%. Many messages were generated from Search Heads indicating blocked Indexing due to full queues. 2) Replication between Indexers would not resolve in a timely manner. Only ONCE in two weeks was the Replication Factor met. Crashing Indexers made "out of sync" pretty common, but HOURS of Replication activity was unable to bring the Indexers back in sync. Most common cause of delayed fixups were "bucket has not rolled" and "Waiting 'target_wait_time' before search factor fixup" (see answers.splunk.com/answers/660917/fixup-status-message-is-waiting-target-wait-time-b for my question on that point). It seems the "target_wait_time" is new in 7.1 - at least I did not see that message after I rolled back to 6.6. Note that version 7.1.0 was applied to the Cluster Master, 2 Search Heads, and 2 Indexers, so all were on the identical version. Also note that the Search Heads did not display any Memory Management issues, and Swap usage remained at 0 (zero!) even after many days of service. Search Heads are 12-core/12GB, Indexers are 12-core/32GB and 12-core/20GB, Cluster Master is 12-core/12GB. Downgrading the installation back to 6.6.2 immediately resolved both issues. Memory Management remained in-bounds (even when utilization reached 100%), and Swap space remained constrained to below 1 GB in the worst case. Index Replication resolved itself in about 1/2 hour. Although both Indexers remained up and healthy through the night, with excellent Memory and Replication statistics in the morning, I still saw one oom-killer incident on one Indexer and three oom-killer incident on the other. I conclude that Memory Management is a HUGE challenge in Splunk 7.1, though I can't say whether that is limited to mult-site Index Clusters. Memory Management is "tolerable" in Splunk 6.6 - but it still has risks. It would be nice for Splunk to issue guidance on how to tame oom-killer so that it does not target the Splunk Processes! Multi-site Index Replication in Splunk 7.1 is unacceptable! I accept that it was exacerbated by the Memory Management / Crashing Indexers, but it should not take hours to sync up a few buckets. Does anyone else have more data that can shed light on resolving these issues? Is it just multi-site clusters? Is 7.1 a memory-hog? Are these bugs? ... View more

In Splunk 7.1.0 with Multi-site Index Cluster, I have a bucket that has been pending fixup for more than 1 hour. The Status message is "Waiting 'target_wait_time' before search factor fixup", but target_wait_time is explicitly set to 160 seconds in etc/system/local/server.conf on the Cluster Master (something slightly different from the default, just to make sure it was used). This is happening on multiple buckets since I upgraded from Splunk 6.6.3. Is this a Bug??? ... View more

Search Head running Splunk 6.5.2 DCN is running Splunk 6.6.1 Splunk Add-on for NetApp Data ONTAP (Splunk_TA_ontap) is version 2.1.5, with AddOn for VMWare version 3.3.2 "over the top" as advised for newest Helpers Removed un-needed SAs and TAs as directed Search Head requires https for Splunk Web (using default splunk certs) Scheduler runs on Search Head (inputs) DCN is Heavy Forwarder and also requires https for Splunk Web (using default splunk certs) DCN has allowRemoteLogin=always in "local" server.conf all Workers are enabled on DCN (inputs) Lots of seemingly random "broken pipes" and other communications issues with the NetApp Appliances, though the Scheduler setup confirms connectivity and authentication is good. Likewise, setup page confirms connectivity to the DCN with correct password (that is not "changeme" ;). Another symptom is that splunk eventually stops on the DCN, though I don't see anything in splunkd.log indicating anything fatal before it simply stops logging! I definitely understand the layout of the Scheduler, DCN, Hydra, etc - there is just a wrinkle somewhere that is not allowing splunk to settle into a stable pattern. Here's an error message from _internal with DEBUG level information included: 2017-06-30 16:16:33,062 ERROR [ta_ontap_collection_scheduler://nidhogg] Problem with hydra scheduler ta_ontap_collection_scheduler://nidhogg: [HTTP 401] Client is not authenticated Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 2102, in run self.node_manifest = self.establishNodeManifest() File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 2019, in establishNodeManifest for node_stanza in node_stanzas: File "/opt/splunk/lib/python2.7/site-packages/splunk/models/base.py", line 133, in _result_iter self._fill_cache() File "/opt/splunk/lib/python2.7/site-packages/splunk/models/base.py", line 140, in _fill_cache self._results_cache.append(self._iter.next()) File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/models.py", line 256, in iterator sessionKey=self._sessionKey) File "/opt/splunk/lib/python2.7/site-packages/splunk/models/base.py", line 297, in get_entities return splunk.entity.getEntities(self.manager.model.resource, unique_key='id', uri=self._uri, namespace=self._namespace, owner=self._owner, **kwargs) File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs) File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True) File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 530, in simpleRequest raise splunk.AuthenticationFailed AuthenticationFailed: [HTTP 401] Client is not authenticated Any ideas from the intrepid initiates of ONTAP? ... View more