Splunk Search

Why does search in fast mode return different results than verbose mode in Splunk Enterprise 7.0.2?

marycordova
SplunkTrust
SplunkTrust

Problem:

  1. search: 1. Search: index=win* EventCode=4624 |userlookup(Account_Name)| table Account_Name name sam eid mail | rename Account_Name as user | search eid!=NONE_FOUND | dedup user name sam eid mail
  2. static time range for explicit comparison: start 8/6/18 13:06:50.000; end 8/6/18 13:21:50.000
  3. fast 13 results; verbose 1257 results
  4. userlookup macro takes in a single attribute and attempts to match it against multiple columns in lookup table: eval $attribute$=lower($attribute$) | lookup ad_users.csv sam as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | lookup ad_users.csv mail as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | lookup ad_users.csv upn as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | eval $attribute$=upper($attribute$) | lookup ad_users.csv eid as $attribute$ OUTPUTNEW name eid sam upn created pwd_set mail tel_ext manager division job department region country | eval $attribute$=lower($attribute$) | eval eid=upper(eid)
@marycordova
1 Solution

marycordova
SplunkTrust
SplunkTrust

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination]
cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

@marycordova

View solution in original post

marycordova
SplunkTrust
SplunkTrust

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination]
cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

@marycordova

Robertoing
Explorer

Hi @marycordova ,

I have a distributed environment and I put this configuration in every SH at path /splunk/etc/system/local, but doesn't work.

Can someone help me to find out the correct solution?

 

Thank to all.

0 Karma

marycordova
SplunkTrust
SplunkTrust

i know this is kind of a lame response but, @Robertoing , are you able to upgrade to version 8?

@marycordova
0 Karma

marycordova
SplunkTrust
SplunkTrust

similar issues on answers:

  1. https://answers.splunk.com/answers/658420/verbose-mode-returns-results-as-expect-but-not-fas.html
  2. https://answers.splunk.com/answers/343834/why-am-i-getting-three-different-results-running-a.html

it would be interesting to see if testing shows this resolves them
it's possible that this specific config doesn't but another option in the stanza would

@marycordova
0 Karma

drfk
New Member

We have the same problem without using a macro in Splunk 6.6.5. The search of type
... | lookup table field1 OUTPUT newfield | lookup table field2 OUTPUTNEW newfield
is (in fast mode) "optimized" to
... | lookup table field2 OUTPUTNEW newfield

Just wanted to confirm that your limits.conf entry solves the problem. Thanks for that!

0 Karma

m2oswald
Explorer

Similar setup to drfk, with no macro but 2 lookups.  Splunk 8.2.2.  Verbose mode gave results, but Fast/Smart modes just resulted in 0's.  Changing the limits.conf file fixed the problem.  Thank you!

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...