Can you please elaborate? I'm having the same problem, but this isn't enough to help me fix my problem. I'm finding the documentation on writing custom commands pretty sparse. ... View more

Here's a way that doesn't require a transpose. <some search here> | eval rowmax=0 | foreach columnname* [ eval rowmax=if('<<FIELD>>' > rowmax, '<<FIELD>>', rowmax) ] You're essentially brute force comparing every column to find the max. ... View more

I also had this problem. This answer directly from a Slack conversation with @micahkemp , who wouldn't let me give him the karma for it. tl;dr: the main fix is this: def prepare(self): self.configuration.required_fields = [self.source_field] here's the full example, for context. from splunklib.searchcommands import dispatch, Configuration, Option from splunklib.searchcommands import StreamingCommand @Configuration(distributed=False) class TestRequiredFieldsCommand(StreamingCommand): source_field = Option(require=True) dest_field = Option(require=True) def prepare(self): self.configuration.required_fields = [self.source_field] def stream(self, records): for record in records: record[self.dest_field] = record[self.source_field] yield record dispatch(TestRequiredFieldsCommand, module_name=__name__) ... View more

yeah, I was unsure on downvoting. I guess I interpreted an out-of-date answer to be "harmful" because I was trying to use the latest recommended technology, and this answer led me down the wrong path and caused me to waste time. I considered that harmful. I can see how other people would disagree. ... View more

I was also looking to answer this question. Much of the documentation that exists is for version 1 of the custom search command protocol, which was deprecated in Splunk 6.4.0. I posted an example here: https://answers.splunk.com/answers/434017/how-to-get-session-key-in-a-search-script-script-s.html#answer-740645 ... View more

I downvoted this post because answer is out of date. it appears that this guidance applies for version 1 of the custom search command protocol, which was deprecated just a couple months before this answer was published. ... View more

The accepted answer is now out-of-date. With the new version 2 of the protocol, use of Intersplunk is deprecated: (as of Splunk 6.4.0):https://docs.splunk.com/Documentation/Splunk/6.4.0/Search/Aboutcustomsearchcommands (as of today) https://docs.splunk.com/Documentation/Splunk/7.2.5/Search/Aboutcustomsearchcommands here is an example that works for me to use the session key to perform a search within a custom command without actually retreiving it myself and adding it as a header: class CustomCommand(StreamingCommand): def stream(self, records): mysearch="search index=_internal" kwargs_create = {'earliest_time':'2019-04-01T12:00:00','latest_time':'2019-04-01:01:00'} job = self.service.jobs.create(mysearch,**kwargs_create) dispatch(IpToUserCommand, sys.argv, sys.stdin, sys.stdout, __name__) Of course, add in all the appropriate error handling. self.service returns a splunklib.client.Service object (https://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.5/searchcommands.html#splunklib.searchcommands.StreamingCommand.service), which already has an authentication token attached. The guidance in @jkat54 post about needing passauth = true in commands.conf still applies ... View more

It would be interesting to test different searches, index segmentation options, indexed field settings, etc. and be able to accurately determine how many I/Os are required to return results. This is almost exactly my use case. Firewall logs have so much repeated data that their cardinality is low (I think... I sometime get those reversed). I'm trying to determine what the best solution to make them faster is: index-time field extraction, data models, segmenters, etc. ... View more

Just found CLONE_SOURCETYPE today in transforms.conf.spec: http://docs.splunk.com/Documentation/Splunk/latest/admin/Transformsconf Sounds like it might be what you need (see excerpts below): CLONE_SOURCETYPE = <string> * If CLONE_SOURCETYPE is used as part of a transform, the transform will create a modified duplicate event, for all events that the transform is applied to via normal props.conf rules. * Use this feature if you need to store both the original and a modified form of the data in your system, or if you want to send the original and a modified form to different outbound systems. * A typical example would be to retain sensitive information according to one policy and a version with the sensitive information removed according to another policy. For example, some events may have data that you must retain for 30 days (such as personally identifying information) and only 30 days with restricted access, but you need that event retained without the sensitive data for a longer time with wider access. Then in the examples: [hide-ip-address] # Make a clone of an event with the sourcetype masked_ip_address. The clone # will be modified; its text changed to mask the ip address. # The cloned event will be further processed by index-time transforms and # SEDCMD expressions according to its new sourcetype. # In most scenarios an additional transform would be used to direct the # masked_ip_address event to a different index than the original data. REGEX = ^(.*?)src=\d+\.\d+\.\d+\.\d+(.*)$ FORMAT = $1src=XXXXX$2 DEST_KEY = _raw CLONE_SOURCETYPE = masked_ip_addresses ... View more

I have the same problem. Here's the very basic lookup script I've created (external_lookup.py) import logging logging.basicConfig(filename='/tmp/splunk-external_lookup-hello.log',level=logging.DEBUG) logging.debug("foo") If I put it in $SPLUNK_HOME/etc/apps/<app_name>/bin , I get the error Could not find 'external_lookup.py'. It is required for lookup 'external_lookup'. But if I put it in $SPLUNK_HOME/etc/apps/<app_name>/bin it works fine. ... View more

Two ways: First way is to save the search, and give it a name (e.g. "My saved search"). DO NOT schedule this search. Then create a different scheduled search and use the savedsearch directive: | savedsearch "My saved search" Then the alert e-mail will only contain the above string, not the original query which might be sensitive. Another way, less work, but also less assured. Splunk's alert e-mails only include the up to the first carriage return in your search. So, you can put some innocuous search stuff at the beginng. For instance: index=* index=sensitive key=value | stats count by sensitivefield When you get the alert e-mail, only the first line ("index=*") will be included. ... View more

Turns out this only works for .csv that are not also gzipped. So, if you write out to .csv.gz, this won't work. If you write out to .csv, it will. ... View more

This does not seem very efficient to me. If your bad domains will only occur in a low percentage of answers, you're reading all those event from disk that will eventually be trashed at the end. Compare that to bwooden's answer, where you're acually searching for and returning only those events that contain the bad domain. In my experience, I/O is the most common limiting factor for Splunk Additionally, at the end of this query, all you'd have are the domains that alerted. In my cases, I usually want to see the whole event. ... View more

I had this problem when the following string was somewhere in my search: username= The sendemail.py script was not properly escaping the search string, and that part of my search was actually changing the username field within sendemail.py, causing it to try authenticating to the SMTP server. (You can see the authentication errors in /opt/splunk/var/log/splunk/python.log ). I was able to fix it by using something other than username in my search. ... View more

You might check that your time extractions are correct. If Splunk's interpretation of each event's time is incorrect, that could lead to the duration field being incorrect. ... View more

I'd like to hear more about this REST API Based on this doc: http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTindex It looks like it would only give you the total DB size, not the hot/warm separate from the cold. That would not help in my instance. Am I interpreting that correctly? ... View more

The du -sm solution does not work for me. The result comes in above my maxVolumeDataSizeMB. I have checked for indexes on that path that aren't configured to use the volume. (I had and fixed that problem before, as you can see here: http://splunk-base.splunk.com/answers/47963/mismatch-between-df-and-splunk-size-of-volume) ... View more

Awesome! I didn't realize that the default indexes could be overridden by a normal app. That should solve it for me. Thanks! ... View more

I have the same question, but your solution (using an app) does not solve it. My problem is that I'm using a volumes configuration for all my indexes, using an app pushed to all my indexers. (The app is called "myindexers".) However, the local indexes (e.g. _audit, _internal, etc.) are residing on the same disk as my custom indexes: $ mount /dev/mapper/db_dg-opt on /opt/splunk/var/lib/splunk type ext3 (rw) Since the internal indexes don't use the volumes config, the disk will fill. I've been advised by people on the IRC channel to fix this by putting some overrides in system/local that force the internal indexes to use the volumes config instead of $SPLUNK_DB. That seems workable, but I'd rather do it with the deployment server if possible, rather than hand maintaining that file on my five different indexers. So, in that situation, is the recommendation still to manually maintain system/local/indexes.conf? Also, since this might be of benefit (truncated to just relevant parts): $ /opt/splunk/bin/splunk cmd btool indexes list [_internal] coldPath = $SPLUNK_DB/_internaldb/colddb homePath = $SPLUNK_DB/_internaldb/db thawedPath = $SPLUNK_DB/_internaldb/thaweddb [dev] coldPath = volume:remote/dev/colddb homePath = volume:local/dev/db thawedPath = /opt/splunk/var/lib/splunk/dev/thaweddb [volume:local] maxVolumeDataSizeMB = 745942 path = /opt/splunk/var/lib/splunk [volume:remote] maxVolumeDataSizeMB = 3774873 path = /splunk_cold ... View more

In my case, it was the Splunk built-in indexes that were causing the most pain (specifically audit and summarydb). Thank you the tip to use 'splunk cmd btool indexes list' to see how all the various config files got merged. That made it much easier to see which indexes were and weren't using my volumes configuration. ... View more