Time difference between log time and time in log line itself (NOT a timezone issue)?

[tbessie]

In my company's Splunk server, when I do a search, I usually see a difference in time between the "Time" column and the "Event" column for each log entry.  An example:

Time: 10/21/24 11:06:37.000 AM
Event: 2024-10-21 11:31:59,232 priority=WARN  ...

Why would the Time column have 11:06:37 but the Event field (the actual logged data) show 11:31:59,232 

[PickleRick]

Ok, the only thing we know for sure is that for this particular event timestamp has not been extracted from the event itself. There can be several reasons for it:

1) Props for this sourcetype, source or host specify assuming ingestion time, not the event time

2) Timestamp format for extraction is wrongly defined and doesn't match the event

3) The event is ingested with a method bypassing timestamp extraction (HEC /event endpoint)

4) Timestamp has been extracted but was out of limits so Splunk assumed timestamp from previous event (but that's relatively unlikely, you'd probably either see many events with the same timestamp or mostly well-extracted time and single exceptions). This can be connected with 2).

5) You have another timestamp within your event which Splunk extracts time from (but I suppose you'd notice that).

Usually the most probable causes are 2, 1 and 3 (in order of frequency).

[isoutamo]

As others already said, it's obviously that timestamp extractions are not working correctly. 

If you could get someone to check this from MC (Monitoring Console) side there should be an answer for a reason.

Settings -> MC

Indexing -> Inputs -> Data Quality

There are some selections to try to find errors. Then just click those error counts and it will open you query which shows more information about that issue. You could also modify that query to get more information about that issue.

Until you can get someone to look those it's not possible to be sure the real reason which a behind this.

r. Ismo

[gcusello]

Hi @tbessie ,

as also @sainag_splunk said, maybe there's a timestamp extraction error.

Could you share some sample of your events and the props.conf related to the sourcetype of these events?

Ciao.

Giuseppe

[tbessie]

Unfortunately, I am not the manager of our Splunk installation (and have no access to it), so I can't provide any info about our setup, config files, etc.

I'll see if I can get that info to you from one of our ops folks.

- Tim

[sainag_splunk]

@tbessie Hello, How are you receiving this data UF or HF? Do you have any TIME format settings on your props.conf ?

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Configuretimestamprecognition#Syntax_o...

I would validate Splunk time parsing configurations first.

Did you validate if the indexer and the source system might have misaligned clocks, I have seen inaccurate search results with misalignment ?

If this Helps, Please Upvote.

[tbessie]

Unfortunately, I am not the manager of our Splunk installation (and have no access to it), so I can't provide any info about our setup, config files, etc.

I'll see if I can get that info to you from one of our ops folks.

- Tim