I have a custom streaming command which takes a long time to startup (120 seconds) as it loads a cache. I stream usually about 100'000 events to this command in one invocation. This works fine except since we changed it to use python 3. The same Splunk query which streams about 100'000 events to this custom command results in the command being called several times with junks of 10'000 events which makes it obviously very inefficient. Its a distributed environment with single search head and a 100 node dual site indexer cluster. The search head find normally about 2'000'000 events which are distributed to 20 indexers, resulting in this 100'000 events per call on the indexer. Any idea why this junks are spread over several calls to the custom command? ... View more

I am trying to setup a federated index, on a federated search head, but i am only able to select an index as the remote dataset. the drop down for dataset type does not offer any other option. How do i have to configure the dataset on the remote search head in order to be able to use it on the federated search head. Bot systems are clustered search heads running Splunk enterprise 8.2.2 ... View more

The following search does not produce any results: index=* earliest="04/19/2022:15:00:00" latest="04/19/2022:17:00:00" | fields _time, index, sourcetype | head 1 | eval mail=[| makeresults | eval mail="\"abc@cde.com\"" | return $mail] The search and the sub search produce each one result. The search returns one result as expected when the  the earliest and latest option in the base search are omitted. ... View more

I have written a generating custom command which generates just one event containing the session token and the search head it runs. This two parameters are then passed to another custom command which runs distributed on the indexers and needs them to call back to the search head. I am on Splunk Enterprise, Version: 8.0.3 with Splunk Python SDK 1.6.12, single search head and 96 node indexer cluster. The python script of the command starts with   from __future__ import absolute_import, division, print_function, unicode_literals import os,sys import time splunkhome = os.environ['SPLUNK_HOME'] sys.path.append(os.path.join(splunkhome, 'etc', 'apps', 'searchcommands_app', 'lib')) from splunklib.searchcommands import dispatch, GeneratingCommand, Configuration, Option, validators @Configuration(streaming=True, local=True) class callbackconnection(GeneratingCommand): .......     The definition of the command in commands.conf is:   [callbackconnection] filename = callbackconnection.py # python.version=python3 chunked = true     This command runs fine while using python 2.x but as soon as python.version=python3 is active in the definition of the command in commands.conf, the command gets distributed to the indexers an returns 97 results. I tried various  @configuration(....) settings, but with python 3 the command is always distributed to the indexers. ... View more

I have an app which included a custom command which in turn has to cache some information on the indexer it runs. What is the best location to store this file, is $SPLUNK_HOME/etc/apps/<appname>/lookups/<my_cache_file_name>.cache sensible? What will the indexer cluster do with this files? Just in case you wonder, the caches are to large to distribute them with the replication bundle as a lookup table.  They are filled by reading the kvstore from the calling search head with REST calls. ... View more

I have a user which needs to be able to write one specific lookup table which has to be shared globally. I have to control with the permission settings on the lookup  as I have several users/roles where each role has to grant write access to a different lookup table. What i have observed so far: in order to be able to write a lookup table with | outputlookup xxxx.csv, the user needs the capability output_file. This capability is be default granted trough the predefined user role. With the capability output_file, the permission configured in local.meta is ignored,  outputlookup writes happily any lookup file, regardless of the permission, even files which don't exist.   What i am missing here? ... View more

I am implementing a streaming distributed custom command in python which needs to access the kv store on the search head where the command has been issued. The kv stores are to large to distribute them to the indexers and simple lookups are to slow. To connect to the kv store with a REST call, I am using the to session_key authenticate. :   kwargs={} kwargs["owner"] = "nobody" kwargs["app"] = "search" kwargs["host"] = "<hostname>" self.headers = {'output_mode': 'json', 'count':10000} self.connection client.connect(token=self._metadata.searchinfo.session_key,**kwargs)   from then on I access the kvstore with kvs = self.connection.kvstore[.....] and result= kvs.data.query(........) But the instance of the executed command on an indexer fails with a login failure as it gets  the session_key from the local session on the indexer which is not valid to connect to the search head. Is it possible to execute a part of the command on the search head before it gets distribute and pass its result (the session_key) to all distributed executed instances? ... View more