Splunk Search

Why are my events not in time order?

New Member

We just upgraded our Splunk server to version 7.0. I created a query that has a time range Between 05/19/2018 04:28:00.000 and 05/19/2018 08:47:00.000. I list 50 events per page. I navigate through pages and I see events in random order. On page 17 (page with oldest events) I see events with these times in this order:
5/19/18 6:11:09.115 AM
5/19/18 5:35:07.463 AM
5/19/18 5:31:00.510 AM
5/19/18 6:08:27.757 AM
5/19/18 6:08:27.753 AM
5/19/18 5:31:00.510 AM
and so on....

There are 2 problems, 1 is that they are not in expected order and 2 the oldest events should have a time close to 05/19/2018 04:28:00.000.

What is going on here?

0 Karma

Splunk Employee
Splunk Employee

This reads like SPL-154973 actually, fixed in 7.1.3+
http://docs.splunk.com/Documentation/Splunk/7.1.3/ReleaseNotes/Fixedissues

Upgrade SH and IDX to 7.1.4+ (can't recommend to upgrade to 7.1.3 for other issues).

0 Karma

New Member

Hi chanfoli, we have a clustered deployment with a single search head and recently upgraded to 7.1.0.
* single SH with distributed search enabled
* clustered indexers

0 Karma

Builder

Thanks for the reply. I also found a question alluding to similar symptoms from another customer from the beginning of the month using the 7.1.x tag - https://answers.splunk.com/answers/655529/search-returning-duplicatedwrong-results-after-upg.html

0 Karma

New Member

Yes, seshi answered for me. I thought we had version 7.0 but seshi did the upgrade so he knows best.

0 Karma

Builder

I have a support case open with what sounds like similar behavior in 7.1.0 - This is with a SH and Indexer cluster, we also notice more strangeness when selecting time ranges on the timeline, it does not properly bound the earliest and latest events and sometimes duplicate events are seen. I am curious about your deployment type, i.e. is it a SH cluster or single SH Are you searching against in indexer cluster or single indexer, and if it is a cluster is it mutli-site?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!