In order to organised things on a search head cluster for various teams/permissions we've been setting up applications on the deployment server with permissions. The idea is that teams would have permissions to deploy searches and evaluations into their own team applications to keep things organised and separated from other teams (or core apps we deploy). One thing we've run into is that if a search is pushed by the deployment server, you can modify it, schedule it, and it runs fine. You can not delete it. If you create a NEW scheduled search in the application from one of the search heads, you can run the search manually but if you save it the date of the next scheduled execution is displayed for about 10 seconds, then set to 'none'. Regardless of what we change the search to it never automatically executes. Does anyone have an idea of what we might be doing wrong? Edit: I just tested the search app, and the result is the same. So any scheduled jobs not provided by the deployer to the cluster are never scheduled. ... View more

I have a query where at the end I specify "| collect index=foo sourcetype=bar" and the results go into that index on a standalone search head. When I do the same on the cluster, it does not go to the index. Additionally the search head complains that it received an event for an unconfigured/disabled/deleted index=foo like it is attempting to write the data locally. What do I need to do to the search head cluster so it targets the indexers when writing summary indexes instead of locally? ... View more

Currently I'm doing an extraction on a log file like so: [AUDIT_PARSE] REGEX = \x5b[^\x5d]+\x5d\s+(\w+)\s+(?:\x7b([^\x7d]+)\x7d){0,1}\x2d\s+(.*) FORMAT = level::$1 log_source::$2 message::$3 One of the extracted fields you see is 'message', which itself has a number of key value pairs. I then want to take the extracted 'message', and parse those pairs. [AUDIT_MESSAGE_PARSE] REGEX = \s*(\S+)\s*\x3a\s*((?:[^\x7c\x5d])+?)\s*(?:\x7c|$) SOURCE_KEY = field:message FORMAT = $1::$2 One of those key value extractions has another field I want parsed, so I do another one on that field: [INITIATOR_PARSE] SOURCE_KEY = field:Initiator REGEX = ^(.*?)\x2f([^\x40]+)(?:\x40(.+)){0,1}$ FORMAT = SSO_Organization::$1 SSO_User::$2 SSO_Domain::$3 I put them all together with the following in props: TRANSFORMS-audit = AUDIT_PARSE, AUDIT_MESSAGE_PARSE, INITIATOR_PARSE Right now AUDIT_PARSE runs and works, but the remainder does not work. If i remove the SOURCE_KEY it 'runs', but it will run in _raw instead of the pre-extracted field. Is this a splunk limitation where I need to do this at Index time, or am I missing something? Thanks! ... View more