Why are search results different when running a search in the Search app versus a dashboard panel?

[splunked38]

Hi,

I have a search as a dashboard panel.

When I execute the search on the dashboard, the result is incorrect.

What's interesting is:

• If I refresh the panel, the result is still incorrect
• when I 'open the search' from the dashboard panel, it's still incorrect
• When hit the search button after the 'open the search' the result is correct.

Unfortunately, I can't post up the search.

Notes:

• At no stage was the search changed
• The time selector was not changed
• The search was working in Splunk <6.3.0
• If I dissect the search, the individual components return the expected result

I'd be interested as to what I can do to check to see where the problem could be.

[splunked38]

I've 'narrowed down' the problem:
the search in the dashboard panel looks like this:

search xxx | stats count(Name) as search1
| eval search2=[search yyy | stats count(Name) as search2| rename search2 as query]
| table search1, search2

search1 is ok, search2 produces the wrong result but if I did this in the dashboard panel:

search yyy  | stats count(Name) as search2 | table search2

The correct result comes up.

Just to re-iterate:
if I go to the dashboard panel, hit 'open in search', the panel search will appear, I hit 'search', and the correct result appears. Because of this, I'm leaning away from the fact that the syntax is the issue. Something about the way the dash executes the search provides incorrect results.

Before you ask, the reason why I need the two searches in the one dash is because I'm trying to get a percentage from the two searches:

search xxx | stats count(Name) as search1
| eval search2=[search yyy | stats count(Name) as search2| rename search2 as query]
| eval percent=round((search2/search1)*100,2).%

[Sebastian2]

I'm very new to Splunk (so my answer may be stupid) - but the first thing coming to my mind is the app context. Is the normal search maybe performed in another app? Do you have anything configured, like transforms combined with auto lookups which affect your search in any way?

[splunked38]

Nope, dash is in search.

If it was app context, why would the second attempt in search be different, I would expect it to be the same

[sundareshr]

Are you using sub-search? If the sub-search reaches the limit, it will return 0 results, thereby affecting the results of the main search.

[sundareshr]

If you look at the job inspector after you run the search in the main search window (after it returns results successfully), is there anything re: results being truncated? If there is, that is the reason dashboard results are inaccurate.

Something else to look for, when you hit "open in search" in dashboard panel, BEFORE you hit the search button, do you see the results? Go look at the "normalizedSearch" in the job inspector window. Does that look correct?

One more place to check - Run the search in dashboard and search window in quick successing. Go to "Activity->Jobs. Compare the Events count from all 4 activities listed (2 for main search, 2 for subsearch). Are they the same?

In my experience, if the result for a sub-search is truncated, the results in dashboard is not the same as what you see in the main search window. Don't know why, just seen this happen.

The other thing to verify is permissions and app context... do those look correct?

[lmonahan]

Super helpful checklist, thanks!  Also will include a link to this answer as base queries with non-transforming results have boundary limits. I think if you're getting inconsistent results its because you're over a limit.

https://community.splunk.com/t5/Splunk-Search/Post-processing-gives-incorrect-results/m-p/522520

[splunked38]

I compared the normalised searches between the two and they are both the same.

Both the 'good' and the 'bad' search are truncated.

Perms and app context are both accessible within the application.

[splunked38]

ok, that's understandable

however, the search string returns the correct result when I manually copy and paste it into search.