Splunk Dev

Ordering Events that arrive with the same timestamp but are out of order

x213217
Explorer

I have a few instances where I will get status events for when jobs are running very quickly and appear as the same timestamp in splunk. When this happens I will get a RUNNING status event after a SUCCESS event, which in fact should be reversed. I am doing processing to get the latest status for certain jobs and this causes a problem with that.

Here is an example below, as you see the two events have an identical timestamp but have been pulled in in reverse order. How do i properly get the latest event when the timestamp is shared like this?

alt text

Tags (1)
0 Karma

sdchakraborty
Contributor

what about sorting by index time? This is how you get index time even though your _time is same but indextime will not be same.

| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")
| sort - indextime

dottey
New Member

This did work for me. I wonder if it would "break" if we ever have to backfill data, though.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Looks like you need to add ms to your logging for better granularity and correct ordering. Your jobs are running and completing in under a second. Nothing Splunk can do to fix this

0 Karma

Vijeta
Influencer

You can |sort - _time statusCode

0 Karma

x213217
Explorer

Unfortunately cannot use that field to sort as for events for Starting Running & Success statuses...these have statusCodes of 3, 1, 4 respectively so the order does not match an increasing pattern

0 Karma

Vijeta
Influencer

you can assign your own values, using eval and if and sort based on new field.

eval status=if(statusCode=1,P2,if(statusCode=3,P1, if(statusCode=4,P4)))
0 Karma
Get Updates on the Splunk Community!

Automatic Discovery Part 1: What is Automatic Discovery in Splunk Observability Cloud ...

If you’ve ever deployed a new database cluster, spun up a caching layer, or added a load balancer, you know it ...

Real-Time Fraud Detection: How Splunk Dashboards Protect Financial Institutions

Financial fraud isn't slowing down. If anything, it's getting more sophisticated. Account takeovers, credit ...

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...