Splunk Search

Why does search in fast mode return different results than verbose mode in Splunk Enterprise 7.0.2?

marycordova
SplunkTrust
SplunkTrust

Problem:

  1. search: 1. Search: index=win* EventCode=4624 |userlookup(Account_Name)| table Account_Name name sam eid mail | rename Account_Name as user | search eid!=NONE_FOUND | dedup user name sam eid mail
  2. static time range for explicit comparison: start 8/6/18 13:06:50.000; end 8/6/18 13:21:50.000
  3. fast 13 results; verbose 1257 results
  4. userlookup macro takes in a single attribute and attempts to match it against multiple columns in lookup table: eval $attribute$=lower($attribute$) | lookup ad_users.csv sam as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | lookup ad_users.csv mail as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | lookup ad_users.csv upn as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | eval $attribute$=upper($attribute$) | lookup ad_users.csv eid as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | eval $attribute$=lower($attribute$) | eval eid=upper(eid)
@marycordova
1 Solution

marycordova
SplunkTrust
SplunkTrust

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination]
cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

@marycordova

View solution in original post

marycordova
SplunkTrust
SplunkTrust

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination]
cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

@marycordova

Robertoing
Explorer

Hi @marycordova ,

I have a distributed environment and I put this configuration in every SH at path /splunk/etc/system/local, but doesn't work.

Can someone help me to find out the correct solution?

 

Thank to all.

0 Karma

marycordova
SplunkTrust
SplunkTrust

i know this is kind of a lame response but, @Robertoing , are you able to upgrade to version 8?

@marycordova
0 Karma

marycordova
SplunkTrust
SplunkTrust

similar issues on answers:

  1. https://answers.splunk.com/answers/658420/verbose-mode-returns-results-as-expect-but-not-fas.html
  2. https://answers.splunk.com/answers/343834/why-am-i-getting-three-different-results-running-a.html

it would be interesting to see if testing shows this resolves them
it's possible that this specific config doesn't but another option in the stanza would

@marycordova
0 Karma

drfk
New Member

We have the same problem without using a macro in Splunk 6.6.5. The search of type
... | lookup table field1 OUTPUT newfield | lookup table field2 OUTPUTNEW newfield
is (in fast mode) "optimized" to
... | lookup table field2 OUTPUTNEW newfield

Just wanted to confirm that your limits.conf entry solves the problem. Thanks for that!

0 Karma

m2oswald
Explorer

Similar setup to drfk, with no macro but 2 lookups.  Splunk 8.2.2.  Verbose mode gave results, but Fast/Smart modes just resulted in 0's.  Changing the limits.conf file fixed the problem.  Thank you!

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...