Has anyone else got inconsistent results in Splunk?

[joe06031990]

Hi,

on our Splunk instance I have set a report using a time chart with a span of 1h and time frame of a day and the report is scheduled to run every hour however each time the report runs it shows different results. Just wondered if anyone has seen this before?

thanks,

joe

[bowesmana]

What is your time range set to, i.e. what is the exact earliest/latest in the search definition.

If you have your 'end' time as now, then it will search up to now, so naturally each hour will have different results.

When you say time frame of a day, do you mean 24h.

Can you expand on what you mean by 'different results'. In what way?

[joe06031990]

Hi ,

the timeframe is set to today and the span in the time chart is 1 hour.

sometime the volume is lower or higher from the same hour.

[bowesmana]

So, at 10am it runs it gives 10 values for the first 10 hours and at 11 am you have 11 values, and are you saying that ANY of the first 10 can have different values or just the value for 10am?

What is the 'ending' time of the search in 'Today'? Is it now or @h 

If it's now, it will be somewhat vague, as it may not contain events that are being indexed at that time, or events that maybe arrive one or two minutes after the search has run, but which have slightly earlier times.

One way to see if you have event 'lag' is to look at _indextime field to see how much difference there is between that and _time. 

If _time is some time before _indextime, you have lag

[ITWhisperer]

In what way are they different?

Have a look at the job inspector to see how many events are processed at each stage.

[joe06031990 · Accepted solution]

Looks like there was a fault with two of the search head nodes.