Why is status superimposed on our logs?

[ArtistOfXtreme]

Hello Guys!

Is my first post so sorry if the title is not as specific as it should be

Look, we have an order tracking report here

The first status is label created at 10:02

Later, a new status "arrived_at_facility" is added, and even tough that's the latest one. "Label_created" is superimposed 

And this continues on and on, the tracking statuses are arriving as normal, but label_created is continued being moved as the latest one.

So our tracking report always takes "label_created" as the latest status, instead of something else as "in_transit" 

Any ideas of what could be wrong with our logs?

Thanks in advance guys. Any additional info you can need, ask away 🙂

[ITWhisperer]

Is this one event (in Splunk terms), i.e. are all the tracking details rewritten to the log whenever there are more details?

If so, you can take the most recent event, with all the details in, break it up into separate splunk events, with the appropriate adjustments to the _time field to get a timeline of tracking detail events

[bowesmana]

Looks like the basic problem is to do with your event timestamping.

Your label created event has the datetime field as 2022-11-09T10:02:42Z, which is a UTC time (because it has Z on the end).

Then the subsequent events have 

arrived_at_facility: 2022-11-08T17:14:00Z
unknown: 2022-11-09T00:34:00Z
departed_origin_facility: 2022-11-08T23:34:00Z

and these are all UTC times, so if Splunk is configured to extract the event time from this field, then your events appear to be all out of sync, so as Splunk works on event "_time", the label_created will always be latest.

You need to address the time ingestion timestamping - that will depend on how your data is coming in and the systems that are producing it.

If you cannot address this, then you can treat the "_indextime" field as _time and do something like

my data search
| eval _time=_indextime
more search...

but this is not a good practice, unless you have no other option - it will not always be guaranteed to be correct .

If the generating systems are in different time zones, then they appear to be putting incorrect times into the data .

[richgalloway]

Are the logs being produced this way by the source system or is this produced by Splunk?  If the latter then what is the SPL that produced the output?

FTR. the label_created event is NOT the latest status, it's just last on the list.  Note the datetime fields.  By default, Splunk displays events in reverse time order so the most recent event is first, not last.