In 6.5.0 Splunk added a bunch of search optimizations, see http://conf.splunk.com/files/2016/recordings/optimized-search-optimization.mp4 / http://conf.splunk.com/files/2016/slides/optimized-search-optimization.pdf for details on what was added.
Sadly, one of those optimizations triggers a bug in SPL. The optimizer would turn this
index=_internal | where user="admin"
where is case sensitive,
search is not. However, if there is a calculated field present for
user, the second search is turned into this lispy expression:
[ AND index::_internal admin case ]
That will silently return incorrect search results.
If you have a field that's not based on indexed tokens, e.g. extracting part of a word, you might search for it like this:
index=_internal | search my_field="value"
The optimizer will again turn it into this:
index=_internal my_field="value" [ AND index::_internal value ]
That will again silently return incorrect search results because there is no indexed token
Note: Regardless of the optimizer changing the results, you should usually use fields.conf to properly declare "this field is not an indexed token", then the optimized results will be correct again and you won't have to use the piped
Hence if you're running 6.5.0 already, you should disable that part of the optimizer in limits.conf:
[search_optimization::predicate_merge] enabled = false
Setting that will give you correct results again.
Case references for Splunkers: 408188, 408195
The first bug (calculated fields) has been fixed in 6.5.1, it seems. IMHO you can now use predicate_merge fairly safely.
While the second bug (field not based on indexed tokens) is still there, it's much harder to trigger and works well together with the
noop workaround... plus, you should just define fields.conf 🙂
The search-optimizer runs at the initial comprehension of the search string, so should only be relevant or be used on a search head. As always it may be useful to bring search config to indexers just for the case of troubleshooting by launching searches directly on an indexer, but that's very much an edge-case scenario.
This is a difficult one to answer 😜
I should mention that if for some reason you want to disable on a search-by-search basis, you can by adding a command to the search