Why is the same search producing different results on same dashboard?

[jbassi1]

I have created a dashboard with two separate graphs one which counts the total number of calls made to the hosts and the second one will show the number of errors

Currently, I have created both graphs exactly the same using the same query on both, later on, I will refine the second search so it filters on ERRORs. I expected both graphs to show the same result but they don't the search I am using is as per below:

cs_dataowner_id="ICTO-18172" cs_stage="$environment$"|search cs_component_id="$domain$" |search source="$callType$"|search source="$callMethod$" |timechart span=15m count as calls by host

Does anyone know why they would show different resultset?

[jbassi1]

Not sure if I understand if I use index I will get consistent results ? If so can you advise how I do this

I want both charts to work against the same time range selected by user on the dashboard I have a Time object from which user selects how far back they want to search. For both charts the "Time Range" property is mapped to the value set in that Time field which it is used in the search. Is that not correct way of doing it ?

I actually will eventually have about 5 charts on this dashboard all needing to be working of the same time range selected by the user

[gcusello]

Hi jbassi1,
at first you don't need to use more search commands:

cs_dataowner_id="ICTO-18172" cs_stage="$environment$"|search cs_component_id="$domain$" source="$callType$" source="$callMethod$" 
| timechart span=15m count as calls by host

Then I suggest to use always the index in searches.

Anyway, try to execute your searches using as time frame yesterday or last hour not latest=now, because if you use now, you could have some difference because in the meantime there could be indexed other logs.

At least, if you have more panels that uses the same main search, you could use the Post Process Search method that uses less machine resources.

Ciao.
Giuseppe

[gcusello]

Hi jbassi1,
sorry if I wasn't clear: I hint to use index only to have more performant searches not for your problem: because if you don't use it, your search will search on all the indexes in the default path instead the only you want in your search.

Anyway, to map all the panels to the same time frame defined in a time input is a correct approach, but, to debug your problem, I hint to use a different time frame (e.g. yesterday or last hour, but without latest=now) to check if in this case you have always different results in your panels or not?
If you have the same results it means that the difference is related to new events indexed during search execution, if you have different results, there's a different problem.

The third hint is to explore the use of Post Process Searches hat gives you a more performant dashboard, you can find information about this in docs and in Splunk Dashboard Examples App.

Ciao.
Giuseppe