Splunk Returning different Stats for same time range.

[djreschke]

I have a scheduled report that runs monthly for the previous month. It runs a cron job 00 08 1 * *. 

I need to go back an validate the report but when i run the raw search i get different results:

Search: index=x sourcetype="xxx" source="*xxx*" group=*-xxx* msg="*assigned to session"
| stats max(_time) as last_login by user, group
| table user group last_login
| convert ctime(last_login)
| rename user as User group as Group last_login as "Last Login"

Results count from the scheduled report Tuesday, September 1, 2020 8:01 AM:  15754 results

results from the ad-hoc search ran this morning 9/8/2020: 15748

I understand this is a small difference but it doesn't make sense why i would have less results now, verses 7 days ago.

Thanks everyone in advance for the help.

[thambisetty]

Getting more results makes sense because there could be delay in indexing of events or timestamp issues. For example if you are taking report for july month on 1st of August there could be 31st of july events indexed late. But getting less number of events is strange. 

[thambisetty]

Can you verify info_max_time and info_min_time for both the searches using _audit logs. May be you can find some clue there?

[isoutamo]

What are your earliest and latest parameters? And also retention time for that index?
r. Ismo

[djreschke]

-1mon@mon - earliest and latest is @mon. and rentention time is 1 year. 

[isoutamo]

When you are running that ad-hoc several times (with some time between queries) are you getting all times the same result? Is there any errors/warnings on job inspector? How about if you change max to latest or latest_time?

[djreschke]

I am getting the same results every time for the ad-hoc searches, no errors, but i am not getting the results from the scheduled search that ran last week. I have not changed the search, to use latest yet. I am trying to understand why the difference. the only change that was done after the schedule search ran was updating the following on the Cluster master. 

[httpServer]
maxThreads=100000
maxSockets=50000

[isoutamo]

So was the scheduled search without latest=@mon? If it is without that probably explain the difference. You could test it with putting latest=“09/01/2020:08:01:00”

[djreschke]

The scheduled search uses the same time inputs for earliest and latest. I ran a different search for the month of july and I am getting more results now verses the scheduled report that ran for that month. 

July Secheduled report results: 14177 

Ad-Hoc search running from 7/01/2020 00:00:00 to 8/01/2020 00:00:00 results: 14187

[thambisetty]

to check when events are indexed 

index=yourindex | eval _time=_indextime | stats count

 _indextime will give you time when events are indexed to indexer.

[isoutamo]

You probably have some events which have come later on to indexers, which explain this.

A good way to check this:

...
| eval iTime = _indextime, dTime = iTime - _time 
| eval dTimeHuman = tostring(dTime, "duration")

r. Ismo