topic Re: Why are search results different when running a search in the Search app versus a dashboard panel? in Splunk Search

Why are search results different when running a search in the Search app versus a dashboard panel?

splunked38 — Mon, 16 Nov 2015 12:07:20 GMT

Hi,

I have a search as a dashboard panel.

When I execute the search on the dashboard, the result is incorrect.

What's interesting is:

If I refresh the panel, the result is still incorrect
when I 'open the search' from the dashboard panel, it's still incorrect
When hit the search button after the 'open the search' the result is correct.

Unfortunately, I can't post up the search.

Notes:

At no stage was the search changed
The time selector was not changed
The search was working in Splunk <6.3.0
If I dissect the search, the individual components return the expected result

I'd be interested as to what I can do to check to see where the problem could be.

Re: Why are search results different when running a search in the Search app versus a dashboard panel?

sundareshr — Mon, 16 Nov 2015 13:52:23 GMT

Are you using sub-search? If the sub-search reaches the limit, it will return 0 results, thereby affecting the results of the main search.

Re: Why are search results different when running a search in the Search app versus a dashboard panel?

splunked38 — Mon, 16 Nov 2015 14:02:55 GMT

ok, that's understandable

however, the search string returns the correct result when I manually copy and paste it into search.

Re: Why are search results different when running a search in the Search app versus a dashboard panel?

Sebastian2 — Mon, 16 Nov 2015 14:15:08 GMT

I'm very new to Splunk (so my answer may be stupid) - but the first thing coming to my mind is the app context. Is the normal search maybe performed in another app? Do you have anything configured, like transforms combined with auto lookups which affect your search in any way?

Re: Why are search results different when running a search in the Search app versus a dashboard panel?

splunked38 — Mon, 16 Nov 2015 14:31:10 GMT

I've 'narrowed down' the problem:
the search in the dashboard panel looks like this:

search1 is ok, search2 produces the wrong result but if I did this in the dashboard panel:

search yyy  | stats count(Name) as search2 | table search2

The correct result comes up.

Just to re-iterate:
if I go to the dashboard panel, hit 'open in search', the panel search will appear, I hit 'search', and the correct result appears. Because of this, I'm leaning away from the fact that the syntax is the issue. Something about the way the dash executes the search provides incorrect results.

Before you ask, the reason why I need the two searches in the one dash is because I'm trying to get a percentage from the two searches:

Re: Why are search results different when running a search in the Search app versus a dashboard panel?

sundareshr — Mon, 16 Nov 2015 14:59:32 GMT

If you look at the job inspector after you run the search in the main search window (after it returns results successfully), is there anything re: results being truncated? If there is, that is the reason dashboard results are inaccurate.

Something else to look for, when you hit "open in search" in dashboard panel, BEFORE you hit the search button, do you see the results? Go look at the "normalizedSearch" in the job inspector window. Does that look correct?

One more place to check - Run the search in dashboard and search window in quick successing. Go to "Activity->Jobs. Compare the Events count from all 4 activities listed (2 for main search, 2 for subsearch). Are they the same?

In my experience, if the result for a sub-search is truncated, the results in dashboard is not the same as what you see in the main search window. Don't know why, just seen this happen.

The other thing to verify is permissions and app context... do those look correct?

Re: Why are search results different when running a search in the Search app versus a dashboard panel?

splunked38 — Mon, 16 Nov 2015 15:26:37 GMT

I compared the normalised searches between the two and they are both the same.

Both the 'good' and the 'bad' search are truncated.

Perms and app context are both accessible within the application.

Re: Why are search results different when running a search in the Search app versus a dashboard panel?

splunked38 — Tue, 17 Nov 2015 11:38:25 GMT

Nope, dash is in search.

If it was app context, why would the second attempt in search be different, I would expect it to be the same

Re: Why are search results different when running a search in the Search app versus a dashboard panel?

lmonahan — Sat, 10 Jun 2023 15:59:18 GMT

Super helpful checklist, thanks! Also will include a link to this answer as base queries with non-transforming results have boundary limits. I think if you're getting inconsistent results its because you're over a limit.

https://community.splunk.com/t5/Splunk-Search/Post-processing-gives-incorrect-results/m-p/522520