Splunk Search

Why are users getting different data for the same query?

sloshburch
Splunk Employee
Splunk Employee

I recently overheard someone asking this and I thought it was worth reposting on here for others' benefit.

Essentially, they've got some rather basic search, like index=_internal sourcetype=splunkd source=*splunk.log | stats count with the time selector set to search over the prior day (not last 24 hours). Yet different users were getting different results. This was most pronounced when users were in different offices.

I've seen two common reasons for this, which I'll answer in a moment. Feel free to add to the conversation with your own experiences and any questions for clarification!

Tags (1)
1 Solution

sloshburch
Splunk Employee
Splunk Employee

Turns out the users had different timezones set! So even though the search was the same, the use of searching over "yesterday" was actually different and relative to their respective time zones. Conversely, if they searched over last 24 hours, this absolute would be consistent regardless of timezone.

This is a good time to direct attention back to https://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureuserswithSplunkWeb and get a good grasp on $SPLUNK_HOME/etc/users/<username>/user-prefs/local/user-prefs.conf

View solution in original post

sloshburch
Splunk Employee
Splunk Employee

A parallel, yet different scenario was that the users were using event sampling. The difference in symptoms was that the same user would see different raw results with each run of the same search over the same absolute time period because different events were sampled and returned. So the user thought something was broken, but in reality, just forgot they enabled sampling 🙂

0 Karma

jplumsdaine22
Influencer

Are you testing questions for the new Architect exam?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Ha ha. Nope, just really obsessed with Customer Success 😉

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Turns out the users had different timezones set! So even though the search was the same, the use of searching over "yesterday" was actually different and relative to their respective time zones. Conversely, if they searched over last 24 hours, this absolute would be consistent regardless of timezone.

This is a good time to direct attention back to https://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureuserswithSplunkWeb and get a good grasp on $SPLUNK_HOME/etc/users/<username>/user-prefs/local/user-prefs.conf

hire_vladimir
Explorer

To quickly rule out the effective time range being applied to the search, open job inspector; look at earliestTime and latestTime fields under job properties, these fields contain timezone offset information as part of the timestamp.

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Enhance Security Operations with Automated Threat Analysis in the Splunk EcosystemAre you leveraging ...

Splunk Developers: Go Beyond the Dashboard with These .Conf25 Sessions

  Whether you’re building custom apps, diving into SPL2, or integrating AI and machine learning into your ...

Index This | How do you write 23 only using the number 2?

July 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...