topic Re: Inconsistent results in Splunk in Splunk Search

Has anyone else got inconsistent results in Splunk?

joe06031990 — Thu, 10 Nov 2022 15:27:15 GMT

Hi,

on our Splunk instance I have set a report using a time chart with a span of 1h and time frame of a day and the report is scheduled to run every hour however each time the report runs it shows different results. Just wondered if anyone has seen this before?

thanks,

joe

Re: Inconsistent results in Splunk

ITWhisperer — Wed, 09 Nov 2022 22:34:19 GMT

In what way are they different?

Have a look at the job inspector to see how many events are processed at each stage.

Re: Inconsistent results in Splunk

bowesmana — Wed, 09 Nov 2022 22:35:03 GMT

What is your time range set to, i.e. what is the exact earliest/latest in the search definition.

If you have your 'end' time as now, then it will search up to now, so naturally each hour will have different results.

When you say time frame of a day, do you mean 24h.

Can you expand on what you mean by 'different results'. In what way?

Re: Inconsistent results in Splunk

joe06031990 — Wed, 09 Nov 2022 22:45:37 GMT

Hi ,

the timeframe is set to today and the span in the time chart is 1 hour.

sometime the volume is lower or higher from the same hour.

Re: Inconsistent results in Splunk

bowesmana — Wed, 09 Nov 2022 22:56:50 GMT

So, at 10am it runs it gives 10 values for the first 10 hours and at 11 am you have 11 values, and are you saying that ANY of the first 10 can have different values or just the value for 10am?

What is the 'ending' time of the search in 'Today'? Is it now or @h

If it's now, it will be somewhat vague, as it may not contain events that are being indexed at that time, or events that maybe arrive one or two minutes after the search has run, but which have slightly earlier times.

One way to see if you have event 'lag' is to look at _indextime field to see how much difference there is between that and _time.

If _time is some time before _indextime, you have lag

Re: Inconsistent results in Splunk

joe06031990 — Sun, 13 Nov 2022 10:49:41 GMT

Looks like there was a fault with two of the search head nodes.