Splunk Search

Why does search in fast mode return different results than verbose mode in Splunk Enterprise 7.0.2?

SplunkTrust
SplunkTrust

Problem:

  1. search: 1. Search: index=win* EventCode=4624 |userlookup(Account_Name)| table Account_Name name sam eid mail | rename Account_Name as user | search eid!=NONE_FOUND | dedup user name sam eid mail
  2. static time range for explicit comparison: start 8/6/18 13:06:50.000; end 8/6/18 13:21:50.000
  3. fast 13 results; verbose 1257 results
  4. userlookup macro takes in a single attribute and attempts to match it against multiple columns in lookup table: eval $attribute$=lower($attribute$) | lookup ad_users.csv sam as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | lookup ad_users.csv mail as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | lookup ad_users.csv upn as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | eval $attribute$=upper($attribute$) | lookup ad_users.csv eid as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | eval $attribute$=lower($attribute$) | eval eid=upper(eid)
1 Solution

SplunkTrust
SplunkTrust

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination]
cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

View solution in original post

SplunkTrust
SplunkTrust

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination]
cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

View solution in original post

SplunkTrust
SplunkTrust

similar issues on answers:

  1. https://answers.splunk.com/answers/658420/verbose-mode-returns-results-as-expect-but-not-fas.html
  2. https://answers.splunk.com/answers/343834/why-am-i-getting-three-different-results-running-a.html

it would be interesting to see if testing shows this resolves them
it's possible that this specific config doesn't but another option in the stanza would

0 Karma

New Member

We have the same problem without using a macro in Splunk 6.6.5. The search of type
... | lookup table field1 OUTPUT newfield | lookup table field2 OUTPUTNEW newfield
is (in fast mode) "optimized" to
... | lookup table field2 OUTPUTNEW newfield

Just wanted to confirm that your limits.conf entry solves the problem. Thanks for that!

0 Karma