Splunk Search

Why am I getting three different results running a search using different search modes (Fast, Smart, Verbose)?

gcusello
SplunkTrust
SplunkTrust

Hi all,

I found a very strange behavior related to Search Modes:
- I have an index with many millions of events migrated from ArcSight,
- if I run a search in Fast, Smart, or Verbose Mode, I have three different results!
- Using the same search on a similar index with less buckets (4 instead of 167) I have always the same result in the three Modes.
- Using the same search on the same index with less results (limiting the time period) I have always the same result in the three Modes

So probably the problem is related to a limit in the max number of buckets used in the search.
Someone can suggest to me where I could find the problem (and the solution!).

Thank you in advance.

Bye.

Giuseppe

1 Solution

gcusello
SplunkTrust
SplunkTrust

It's a bug solved in 6.3.4 version.

View solution in original post

marycordova
SplunkTrust
SplunkTrust

https://answers.splunk.com/answers/679070/search-in-fast-mode-comes-back-with-different-resu.html#an...

Splunk JIRA SPL-153269

A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.

Make the following configuration change to limits.conf:
[search_optimization::projection_elimination] cmds_black_list = lookup

There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.

Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.

@marycordova
0 Karma

marycordova
SplunkTrust
SplunkTrust

it would be interesting to see if testing shows this resolves the issue
it's possible that this specific config doesn't but another option in the stanza would

@marycordova
0 Karma

DUThibault
Contributor

In 7.2.6 the cmds_black_list = lookup clause is absent from the [search_optimization::projection_elimination] stanza. Either it is now built-in, or projection_elimination now deals correctly with lookups.

0 Karma

gcusello
SplunkTrust
SplunkTrust

It's a bug solved in 6.3.4 version.

vxsplunk
Explorer

cusello, could you simplify your search to a basic search (for example: index=a host=x sourcetype=y) and leave out transformations, extracted fields and others? Still experiencing different result counts?

I have the same problem with a search that has a result of 3.000.000+ events over 4 hours. Fast Mode is correct (I did a manual count of the log file) and verbose/smart are wrong (larger result count).

0 Karma

gcusello
SplunkTrust
SplunkTrust

The problem you describe is the same I have (the only different is that in Verbose mode I have less events!).

The problem isn't related to the search: I also used a very simple search (index=XXX | stats count by sourcetype) and I found the same problem!
At the same time I found that if I run my search on an equivalent number of events but distributed in a less number of buckets I have the same results in Fast and Verbose Mode.

Splunk Assistance confermed that this is a bug that will be solved in the 6.3.4 version.

Bye.
Giuseppe

0 Karma

vxsplunk
Explorer

Leave out the "stats" (Fast mode handles transforming commands differently and stats is a transforming command++). Still different results?

++) "Only depicts search results as report result tables or visualizations when you run a reporting search (a search that includes transforming commands)." http://docs.splunk.com/Documentation/Splunk/6.3.3/Search/Changethesearchmode

0 Karma

gcusello
SplunkTrust
SplunkTrust

My problem is that in my dashboard I always have transforming commands (the only exception is "table") and opening results in the search form I always have different results between Fast and Verbose Mode (also with "table").
And the correct result is the one in Fast Mode!
Thank you.
Bye.
Giuseppe

0 Karma

bruceclarke
Contributor

I'm having a similar issue. Based on the logs, the verbose search is correct. As a workaround until 6.3.4, I want to run my dashboard panel in verbose mode. Is there a way to do this?

0 Karma

gcusello
SplunkTrust
SplunkTrust

I found that dashboard always use intelligent mode.
I had problems in drilldown to the search page because the search mode I find is the last used by the user.
In other words if the last time I used verbose mode, the search page is open in verbose mode.
Bye.
Giuseppe

0 Karma

bruceclarke
Contributor

If there's no way to force the dashboard to search in verbose mode, then this is a big problem. Dashboards will show incorrect data and could claim tests are passing or pages are loading when there are actually issues.

If you have the case number, please provide it for us to follow. Thanks!

0 Karma

jplumsdaine22
Influencer

Crikey - any chance you can put the bug tracker number here in case someone is searching for it?

Thanks for saying how you resolved the problem!

0 Karma

jplumsdaine22
Influencer

Is you search just index=foo or index=foo some_field=bar ?

If its the latter, then you may not have the same fields available in fast mode and as such you may get different results.

0 Karma

gcusello
SplunkTrust
SplunkTrust

I have a very long search, with extracted fields, regex and calculated fields, but I don't think that the problem are the fields because the correct result is the one obtained in Fast Mode.

Everyway, running the same search on a less time period (less events), I have the same results for Fast, Smart and Detailed Search Mode.

In addition, I have my events in two indexes: events from ArcSight in "GP2-arcsight" index and events received directly from host (the same that before sent events to ArcSight) in "GP2" index.
The first one is divided in 167 buckets and the second one in 4 buckets.
Running the same search on the second index (with about the same number of events) I have the same results for Fast, Smart and Detailed Search Mode.

For these reasons I think that the problem could be the number of buckets that reaches the limit of 300, but I wasn't be able to find where to set this parameter.

Thanks.

Bye.

Giuseppe

0 Karma

jplumsdaine22
Influencer

As far as I know the extra buckets should just slow the search down rather than make it malfunction. But I'm no expert on buckets - someone will have better information than I. Feel free to downvote my answer if it's not applicable 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...