Why are users getting different data for the same query?

[sloshburch]

I recently overheard someone asking this and I thought it was worth reposting on here for others' benefit.

Essentially, they've got some rather basic search, like index=_internal sourcetype=splunkd source=*splunk.log | stats count with the time selector set to search over the prior day (not last 24 hours). Yet different users were getting different results. This was most pronounced when users were in different offices.

I've seen two common reasons for this, which I'll answer in a moment. Feel free to add to the conversation with your own experiences and any questions for clarification!

[sloshburch]

A parallel, yet different scenario was that the users were using event sampling. The difference in symptoms was that the same user would see different raw results with each run of the same search over the same absolute time period because different events were sampled and returned. So the user thought something was broken, but in reality, just forgot they enabled sampling 🙂

[jplumsdaine22]

Are you testing questions for the new Architect exam?

[sloshburch]

Ha ha. Nope, just really obsessed with Customer Success 😉

[sloshburch · Accepted solution]

Turns out the users had different timezones set! So even though the search was the same, the use of searching over "yesterday" was actually different and relative to their respective time zones. Conversely, if they searched over last 24 hours, this absolute would be consistent regardless of timezone.

This is a good time to direct attention back to https://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureuserswithSplunkWeb and get a good grasp on $SPLUNK_HOME/etc/users/<username>/user-prefs/local/user-prefs.conf

[hire_vladimir]

To quickly rule out the effective time range being applied to the search, open job inspector; look at earliestTime and latestTime fields under job properties, these fields contain timezone offset information as part of the timestamp.