Dashboard panel is rendering different value than actual search

[Poojitha]

Hi All,

I have a panel in the  classic dashboard that has pie chart visualisation. Below is the query : 

index="*test" sourcetype=aws:test host=testhost |table lvl msg _time source host tnt | search lvl IN (Error, Warn) source="*testsource*" | chart count BY lvl| sort -count

When I run the query it is showing result 
lvl == warn = 304, error=5 . But in pie chart it is showing different count ->warn=325, error=7

Not getting what is causing  this. Please can anyone help me to know on this. I really appreciate that.

Thanks,
PNV

[ITWhisperer]

What does "When I run the query" mean? Are you copying/rewriting the search in a search window, or are you using the "Open in Search" button on the pie-chart?

Does the timeframe for the results of the search match the time frame you think you are using?

For example:

[Poojitha]

@ITWhisperer  : I tried both the ways. Run manually copy pasting the query as well as running from search icon in the widget. Also , by setting the time window manually. Still there was discrepancy.

[ITWhisperer]

Does the time used in the search match the time you were expecting?

[gcusello · Accepted solution]

Hi @Poojitha ,

two things:

at first put all the search terms in the main search to have a more performant search:

index="*test" sourcetype=aws:test host=testhost lvl IN (Error, Warn) source="*testsource*" 
| stats count BY lvl
| sort -count

second thing: to compare two searches you have to use a defined time frame and never latest=now because in the meantime you could have new events, so run your search in a past timeframe (e.g. like @livehybrid hinted) or previous hour.

Ciao.

Giuseppe

[Poojitha]

@gcusello  sorry for getting back late . 

I tried as you said , it is working fine now ( values are matching now). 

index="*test" sourcetype=aws:test host=testhost lvl IN (Error, Warn) source="*testsource*" 
| table lvl msg _time source host tnt
| stats count by lvl

Please can you help me to know what actually was happening at backend when chart was used  when it supposed to give same result ?  (that is in my original query).  Why that effected dashboard widget ? 
Sharing this knowledge really helps me. 

Thanks in advance,
PNV

[livehybrid]

Hi @Poojitha 

Can you confirm that you are running the search across the exact time frame?
e.g. "Yesterday"

If you run something like "Last 24 hours" then the actual timeframe will be different each time you run it, this would explain why your values are slightly different when running the search versus viewing the dashboard.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[Poojitha]

@livehybrid  : Thanks for the response. 

The time frame is dynamic from time picker in the dashboard. I tried for last 60 mins and expanded time range as well. In all of the cases there is discrepancy. 

[livehybrid]

So @Poojitha  - If you ran the dashboard search now for last 60 minutes, it would search from the start of the minute 60 minutes ago, until now. For example 08:40:00.000 to 09:40:12.000 (Note that "Now" in this case is 09:40:12 - 12 seconds after the start of 09:40).

If you now ran the same search in the Splunk search bar, 10 seconds later (for example) you would be searching 08:40:00.000 to 09:40:22.000 - so, the reason this is interesting is that you may have more counts and more errors in the last 10 seconds.

To verify the counts you will need to run the search over a specific time window in both the dashboard and Splunk Search bar.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[Poojitha]

@livehybrid Thanks for this detailed explanation. This helped me as well.

[Poojitha]

@livehybrid  Thanks for detailed explanation. I understand your point

But when I tried using stats command  it is working fine when I load the query from splunk dashboard widget where it failed with chart. So still not getting what actually went wrong.

index="*test" sourcetype=aws:test host=testhost lvl IN (Error, Warn) source="*testsource*" 
| table lvl msg _time source host tnt
| stats count by lvl

Thanks,
PNV