Why does search in fast mode return different results than verbose mode in Splunk Enterprise 7.0.2?

[marycordova]

Problem:

1. search: 1. Search: index=win* EventCode=4624 |userlookup(Account_Name)| table Account_Name name sam eid mail | rename Account_Name as user | search eid!=NONE_FOUND | dedup user name sam eid mail
2. static time range for explicit comparison: start 8/6/18 13:06:50.000; end 8/6/18 13:21:50.000
3. fast 13 results; verbose 1257 results
4. userlookup macro takes in a single attribute and attempts to match it against multiple columns in lookup table: eval $attribute$=lower($attribute$) | lookup ad_users.csv sam as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | lookup ad_users.csv mail as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | lookup ad_users.csv upn as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | eval $attribute$=upper($attribute$) | lookup ad_users.csv eid as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | eval $attribute$=lower($attribute$) | eval eid=upper(eid)

[marycordova · Accepted solution]

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination]
cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

[rajan_kumar_rai]

Facing the same issue in Splunk Enterprise version  - 8.2.6.1 

Any fix? workaround? please share !!

[Robertoing]

Hi @marycordova ,

I have a distributed environment and I put this configuration in every SH at path /splunk/etc/system/local, but doesn't work.

Can someone help me to find out the correct solution?

Thank to all.

[marycordova]

i know this is kind of a lame response but, @Robertoing , are you able to upgrade to version 8?

[marycordova]

similar issues on answers:

1. https://answers.splunk.com/answers/658420/verbose-mode-returns-results-as-expect-but-not-fas.html
2. https://answers.splunk.com/answers/343834/why-am-i-getting-three-different-results-running-a.html

it would be interesting to see if testing shows this resolves them
it's possible that this specific config doesn't but another option in the stanza would

[drfk]

We have the same problem without using a macro in Splunk 6.6.5. The search of type
... | lookup table field1 OUTPUT newfield | lookup table field2 OUTPUTNEW newfield
is (in fast mode) "optimized" to
... | lookup table field2 OUTPUTNEW newfield

Just wanted to confirm that your limits.conf entry solves the problem. Thanks for that!

[m2oswald]

Similar setup to drfk, with no macro but 2 lookups.  Splunk 8.2.2.  Verbose mode gave results, but Fast/Smart modes just resulted in 0's.  Changing the limits.conf file fixed the problem.  Thank you!