That's correct and is probably how this problem gets introduced most frequently, although it's probably not the only way it can be introduced. Deletion of Enterprise Security correlation searches via the normal Splunk Manager GUI is not supported because a Correlation Search consists of several configuration file artifacts - stanzas in savedsearches.conf and correlationsearches.conf. The normal Splunk Manager page for managing searches is not aware of the other configuration files. So, deleting a Correlation Search via the unapproved mechanism will leave the system in an inconsistent state. All management of Correlation Searches is expected to occur through the configuration pages in the Enterprise Security app - which do not currently support deletion to my knowledge, only disabling. We've rectified the error in our configuration check in the next ES release to more properly reflect the condition - we now warn that you have an "orphaned" stanza in correlationsearches.conf, possibly as a result of an attempt to manage a Correlation Search via the unapproved mechanism. ... View more

I would recommend running these two queries via CURL on your system and appending them to the support ticket if you have already created one. Also feel free to send me the salesforce ticket number directly; my username minus the "_splunk" @splunk.com will get to me. curl -k -u admin https://127.0.0.1:8089/servicesNS/dhorn/DA-ESS-IdentityManagement/saved/searches/Access%20-%20Interactive%20Logon%20by%20a%20Service%20Account%20-%20Rule curl -k -u admin https://127.0.0.1:8089/servicesNS/nobody/DA-ESS-IdentityManagement/saved/searches/Access%20-%20Interactive%20Logon%20by%20a%20Service%20Account%20-%20Rule Also, if a diag hasn't been created for this issue, submit one as we would like to see the configuration files from the system, if possible. ... View more

(Splitting reply in half due to arbitrary character limits) There are probably two errors here: The configuration check is verifying that any correlation searches that use the new "Extreme Search" capabilities packaged with ES are enabled in tandem with their baselining searches. This check should not be erroring out when it encounters a search that it cannot identify. I'll track this as a new bug in the ES project (I am one of the developers on the team). The real error here is that this search: "Access - Interactive Logon by a Service Account - Rule" likely does not exist or cannot be found. Based on the URL being shown in the log, which contains a username ( https://127.0.0.1:8089/servicesNS/**dhorn**/DA-ESS-IdentityManagement/saved/searches) I expect that the error here may be that we are using the "owner" field from the correlation search, and that field is returning a user name. In order to retrieve a Correlation Search without incident, it should probably be using the unscoped name "nobody" here. ... View more

What if you change this: LOOKUP-hq = regionhq name AS country OUTPUTNEW hq AS hq to this: LOOKUP-xhq = regionhq name AS country OUTPUTNEW hq AS hq I believe these are processed alphabetically in order of lookup name. Since LOOKUP-hq > LOOKUP-ip, you may be running into an order-of-operations issue. ... View more

Did you restart Splunk to make the updated log.cfg take effect? I'm not seeing the messages following a restart, but am likely on a different product version. To make the settings take effect immediately, you can also do this: splunk set log-level TransformsExtractionHandler -level ERROR However I don't think that will persist beyond a restart. If you continue to have trouble, I'd suggest opening a support case; there could be other issues at play. ... View more

The warning is harmless (except for consuming disk space and I/O when being written). You can suppress it by setting this in log.cfg: category.TransformsExtractionHandler=ERROR However, you'd lose other warning messages from that category via that solution; caveat emptor. ... View more

Thuan- I did not see that this question had been answered... I've tested this extensively and the max_memtable_bytes setting is unlikely to improve performance so long as you are using the CIDR lookup type, for lookup tables of your size. I think your best bet is to either: a) Use an external Python lookup to perform a binary search. If you are on Windows, the performance of this method will be less substantial than the performance improvement you will see on a Linux system, since the cost of forking Python processes is larger on Windows, in my experience. b) Expand all CIDR subnets in your lookup table into individual IP addresses and use a string-based lookup instead of a CIDR lookup. This will increase your lookup table size substantially, but if the lookup table is likely to be static, you will only incur a one-time cost when the lookup is indexed (assuming max_memtable_bytes is set to a value smaller than the lookup table size), and the lookup should be very fast thereafter. I would only recommend this approach if you are using subnets smaller than /24 in your lookup table; anything larger will probably cause the size of your lookup table to expand too dramatically, and the performance increase you will see from using the string-based lookup will be offset by the size of the lookup table. ... View more

This error is occurring because Enterprise Security contains a "configuration_checker.py" modular input that attempts to alert you when misconfigurations are detected - we attempt to be proactive and alert to conditions that might be causing the application to misbehave. In this case, the intent of the alert is to alert when a scripted input or modular input has exited abnormally. The definition of "abnormally" that we use means "exited with a non-zero exit code". Sometimes, as in this case, this particular check backfires. The error you're seeing is benign and is occurring because the scripted inputs included with the TA-windows add-on use non-zero exit codes even when they exit successfully. This has been corrected in an upcoming version (which of course doesn't help in this instance). If you would like to just get rid of the message, you can disable this input stanza in the Manager: Settings --> Data Inputs --> Configuration Checker --> confcheck_script_errors However, that would disable the exit status checking for ALL scripted and modular inputs on the system. If you'd like a more robust solution, reach out to support and mention this posting, and we can provide a small patch to the configuration_checker.py script which will deal with this in a more intelligent fashion - with the caveat that it wouldn't persist beyond an upgrade. ... View more

Custom fields in asset and/or identity tables were prohibited by design beginning in Enterprise Security 2.2, which contained performance optimizations for asset and identity correlation. However, we now have a patch that provides custom field support for Enterprise Security 2.4. To obtain it you should contact Support to open a case. Provide your exact Splunk and app version information, and we should be able to help. ... View more