topic Re: Splunk Returning different Stats for same time range. in Splunk Search

Splunk Returning different Stats for same time range.

djreschke — Tue, 08 Sep 2020 15:53:18 GMT

I have a scheduled report that runs monthly for the previous month. It runs a cron job 00 08 1 * *.

I need to go back an validate the report but when i run the raw search i get different results:

Search: index=x sourcetype="xxx" source="*xxx*" group=*-xxx* msg="*assigned to session"
| stats max(_time) as last_login by user, group
| table user group last_login
| convert ctime(last_login)
| rename user as User group as Group last_login as "Last Login"

Results count from the scheduled report Tuesday, September 1, 2020 8:01 AM: 15754 results

results from the ad-hoc search ran this morning 9/8/2020: 15748

I understand this is a small difference but it doesn't make sense why i would have less results now, verses 7 days ago.

Thanks everyone in advance for the help.

Re: Splunk Returning different Stats for same time range.

isoutamo — Tue, 08 Sep 2020 18:59:44 GMT

What are your earliest and latest parameters? And also retention time for that index?
r. Ismo

Re: Splunk Returning different Stats for same time range.

djreschke — Tue, 08 Sep 2020 19:15:34 GMT

-1mon@mon - earliest and latest is @mon. and rentention time is 1 year.

Re: Splunk Returning different Stats for same time range.

isoutamo — Tue, 08 Sep 2020 19:19:11 GMT

When you are running that ad-hoc several times (with some time between queries) are you getting all times the same result? Is there any errors/warnings on job inspector? How about if you change max to latest or latest_time?

Re: Splunk Returning different Stats for same time range.

djreschke — Tue, 08 Sep 2020 19:26:47 GMT

I am getting the same results every time for the ad-hoc searches, no errors, but i am not getting the results from the scheduled search that ran last week. I have not changed the search, to use latest yet. I am trying to understand why the difference. the only change that was done after the schedule search ran was updating the following on the Cluster master.

[httpServer]
maxThreads=100000
maxSockets=50000

Re: Splunk Returning different Stats for same time range.

isoutamo — Tue, 08 Sep 2020 19:32:09 GMT

So was the scheduled search without latest=@mon? If it is without that probably explain the difference. You could test it with putting latest=“09/01/2020:08:01:00”

Re: Splunk Returning different Stats for same time range.

thambisetty — Tue, 08 Sep 2020 19:35:21 GMT

Can you verify info_max_time and info_min_time for both the searches using _audit logs. May be you can find some clue there?

Re: Splunk Returning different Stats for same time range.

djreschke — Tue, 08 Sep 2020 19:38:04 GMT

The scheduled search uses the same time inputs for earliest and latest. I ran a different search for the month of july and I am getting more results now verses the scheduled report that ran for that month.

July Secheduled report results: 14177

Ad-Hoc search running from 7/01/2020 00:00:00 to 8/01/2020 00:00:00 results: 14187

Re: Splunk Returning different Stats for same time range.

thambisetty — Tue, 08 Sep 2020 19:41:51 GMT

Getting more results makes sense because there could be delay in indexing of events or timestamp issues. For example if you are taking report for july month on 1st of August there could be 31st of july events indexed late. But getting less number of events is strange.

Re: Splunk Returning different Stats for same time range.

isoutamo — Wed, 09 Sep 2020 06:08:06 GMT

You probably have some events which have come later on to indexers, which explain this.

A good way to check this:

... | eval iTime = _indextime, dTime = iTime - _time | eval dTimeHuman = tostring(dTime, "duration")

r. Ismo

Re: Splunk Returning different Stats for same time range.

thambisetty — Wed, 09 Sep 2020 05:14:49 GMT

to check when events are indexed

index=yourindex | eval _time=_indextime | stats count

_indextime will give you time when events are indexed to indexer.