Why am I getting three different results running a search using different search modes (Fast, Smart, Verbose)?

[gcusello]

Hi all,

I found a very strange behavior related to Search Modes:
- I have an index with many millions of events migrated from ArcSight,
- if I run a search in Fast, Smart, or Verbose Mode, I have three different results!
- Using the same search on a similar index with less buckets (4 instead of 167) I have always the same result in the three Modes.
- Using the same search on the same index with less results (limiting the time period) I have always the same result in the three Modes

So probably the problem is related to a limit in the max number of buckets used in the search.
Someone can suggest to me where I could find the problem (and the solution!).

Thank you in advance.

Bye.

Giuseppe

[marycordova]

https://answers.splunk.com/answers/679070/search-in-fast-mode-comes-back-with-different-resu.html#an...

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination] cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

[marycordova]

it would be interesting to see if testing shows this resolves the issue
it's possible that this specific config doesn't but another option in the stanza would

[DUThibault]

In 7.2.6 the cmds_black_list = lookup clause is absent from the [search_optimization::projection_elimination] stanza. Either it is now built-in, or projection_elimination now deals correctly with lookups.

[gcusello · Accepted solution]

It's a bug solved in 6.3.4 version.

[vxsplunk]

cusello, could you simplify your search to a basic search (for example: index=a host=x sourcetype=y) and leave out transformations, extracted fields and others? Still experiencing different result counts?

I have the same problem with a search that has a result of 3.000.000+ events over 4 hours. Fast Mode is correct (I did a manual count of the log file) and verbose/smart are wrong (larger result count).

[gcusello]

The problem you describe is the same I have (the only different is that in Verbose mode I have less events!).

The problem isn't related to the search: I also used a very simple search (index=XXX | stats count by sourcetype) and I found the same problem!
At the same time I found that if I run my search on an equivalent number of events but distributed in a less number of buckets I have the same results in Fast and Verbose Mode.

Splunk Assistance confermed that this is a bug that will be solved in the 6.3.4 version.

Bye.
Giuseppe

[vxsplunk]

Leave out the "stats" (Fast mode handles transforming commands differently and stats is a transforming command++). Still different results?

++) "Only depicts search results as report result tables or visualizations when you run a reporting search (a search that includes transforming commands)." http://docs.splunk.com/Documentation/Splunk/6.3.3/Search/Changethesearchmode

[gcusello]

My problem is that in my dashboard I always have transforming commands (the only exception is "table") and opening results in the search form I always have different results between Fast and Verbose Mode (also with "table").
And the correct result is the one in Fast Mode!
Thank you.
Bye.
Giuseppe

[rajan_kumar_rai]

Exactly the same issue facing in Splunk Enterprise version 8.2.6.1

Any fix? or workaround? 

[bruceclarke]

I'm having a similar issue. Based on the logs, the verbose search is correct. As a workaround until 6.3.4, I want to run my dashboard panel in verbose mode. Is there a way to do this?

[gcusello]

I found that dashboard always use intelligent mode.
I had problems in drilldown to the search page because the search mode I find is the last used by the user.
In other words if the last time I used verbose mode, the search page is open in verbose mode.
Bye.
Giuseppe

[rajan_kumar_rai]

Hi @gcusello ,

Did you get the solution for this issue? 

I am using Splunk v8.2.6.1 and I am facing the same issue.

Please help if you have any solution for this!

[gcusello]

Hi @rajan_kumar_rai ,

I didn't have experience on this issue after thst (8 years ago).

If you still have this issue, open a case to Splunk Support.

But before update you Splunk becaue I'm not sure that your release is still under maintenance.

Ciao.

Giuseppe

[bruceclarke]

If there's no way to force the dashboard to search in verbose mode, then this is a big problem. Dashboards will show incorrect data and could claim tests are passing or pages are loading when there are actually issues.

If you have the case number, please provide it for us to follow. Thanks!

[jplumsdaine22]

Crikey - any chance you can put the bug tracker number here in case someone is searching for it?

Thanks for saying how you resolved the problem!

[jplumsdaine22]

Is you search just index=foo or index=foo some_field=bar ?

If its the latter, then you may not have the same fields available in fast mode and as such you may get different results.

[gcusello]

I have a very long search, with extracted fields, regex and calculated fields, but I don't think that the problem are the fields because the correct result is the one obtained in Fast Mode.

Everyway, running the same search on a less time period (less events), I have the same results for Fast, Smart and Detailed Search Mode.

In addition, I have my events in two indexes: events from ArcSight in "GP2-arcsight" index and events received directly from host (the same that before sent events to ArcSight) in "GP2" index.
The first one is divided in 167 buckets and the second one in 4 buckets.
Running the same search on the second index (with about the same number of events) I have the same results for Fast, Smart and Detailed Search Mode.

For these reasons I think that the problem could be the number of buckets that reaches the limit of 300, but I wasn't be able to find where to set this parameter.

Thanks.

Bye.

Giuseppe

[jplumsdaine22]

As far as I know the extra buckets should just slow the search down rather than make it malfunction. But I'm no expert on buckets - someone will have better information than I. Feel free to downvote my answer if it's not applicable 🙂