Splunk Search

Why are events not sorting in Chronological Order with a basic search?

rtev
Path Finder

Today, I noticed that, when performing a basic search, the events are not sorted chronologically. Additionally, not all events 'match up' correctly to the timeline.

I have not found any other posts which document this strange behavior.

With a simple | sort _time, the events sort as expected and correlate to the timeline accurately.

The deployment was upgraded from 7.0.2 to 7.1.2 one week ago.

Here's some screenshots that show the behavior:

Events not in Chronological Order
Events not Correlated with Timeline

Does anyone have any ideas how to fix this issue?

1 Solution

Ranazar
Path Finder

Yeah, we noticed the same thing on our side. Opened a ticket with Splunk and they confirmed that it was a known bug:

This behavior has been reported as a bug on a Jira (SPL-154973) document, and has been fixed on version 7.1.3.

And the official workaround is to explicitly sort the events.

View solution in original post

Ranazar
Path Finder

Yeah, we noticed the same thing on our side. Opened a ticket with Splunk and they confirmed that it was a known bug:

This behavior has been reported as a bug on a Jira (SPL-154973) document, and has been fixed on version 7.1.3.

And the official workaround is to explicitly sort the events.

Ranazar
Path Finder

Incidentally (and I'm not sure if this is a symptom of the same issue), I've actually had the search results show the same events multiple times (as though they were different events). As before, explicit sorting fixes this.

0 Karma

rtev
Path Finder

Earlier this morning I noticed some other similar aberrant behaviors related to events showing up multiple times when zooming into different slices of the timeline. Similarly I don't know if it's a symptom of the same issue but it seems likely. I will accept your answer when things get straightened out. Thank you!

0 Karma

rtev
Path Finder

I can't edit my own post... Correction: The version was upgraded to 7.1.2.

0 Karma

sideview
SplunkTrust
SplunkTrust

It's certainly a little strange. In general if you have searchterms | <some transforming command> the order of the events going into the transforming command are not actually guaranteed to be in time order. (Yes it used to be true long long ago, but with distsearch and search-in-separate-process and various parallel bucket things they did, it's no longer always true)

HOWEVER why am I talking about transforming commands? You're seeing this happen in a simple events search. Yes, I am surprised.

I suspect that it's something you don't normally see unless the timestamps on the events are a little different from the actual wall-clock-time when they come into the system? Is there anything else notable about those events whose timestamps are off from the others?

0 Karma

rtev
Path Finder

I should also mention that Transparent Huge Page memory management was disabled one day ago on all (Linux) hosts across the cluster.

0 Karma

rtev
Path Finder

The timestamps are coming in as Unix Epoch time and are extracted correctly. I've checked the difference between the _indextime and _time and there are no events for which the skew is greater than 4-5 seconds. I'd be glad to share a couple of examples of the data that is being ingested and the related props configuration if you think that might shed some light on the matter. Thanks!

0 Karma

rtev
Path Finder

I should mention that the timestamps resolve down to milliseconds...

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...