Getting Data In

Why is Splunk showing a time difference?

Jackinout9
Loves-to-Learn

Hi
We are trying to write the props from couple of days

Issue: splunk showing time difference 4 to 5 hours

logs are coming from one source with multiple time differences ..
example 1.
splunk time 3:48pm, log time 20:48 .
example 2.
splunk time 2:24pm log time 18:24.
time format : 2023-03-10T20:48:11.689534088Z

 

Please let me know if you have any ideas or solutions that could help us out here!

 

Labels (3)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Jackinout9 ,

could you share a sample of events, both correctly and not correctly indexed?

maybe the problem is the presence of another time inside the events and the timestamp isn't correctle read.

Ciao.

Giuseppe

0 Karma

Jackinout9
Loves-to-Learn

Hi @gcusello 

 

I tried with below props

[ <SOURCETYPE NAME> ]

 SHOULD LINEMERGE=false

 LINE BREAKER= ([\r\n]+) 

NO BINARY CHECK=true

 TIME PREFIX="time"

CHARSET=UTF-8 

disabled=false

 

Sample logs :

3/10/23

3:48:11.689 PM

{ [-]

log: [20:48:11] [<unknown>][9f9835b5][ExtensionHostConnection] <417845> Extension Host Process exited with code: 0, signal: null.

 

stream: stdout

 

time: 2023-03-10T20:48:11.689534088Z

}

Show as raw text

3/13/23

2:24:20.526 PM

{ [-]

log: [18:24:20] [unknown][7fafc71d][ManagementConnection] New connection

 

established.

 

stream: stdout

 

time: 2023-03-13T18:24:20, 5264506632

}

Show as raw text

 

And also I attached sample pic

0 Karma

PickleRick
SplunkTrust
SplunkTrust

One thing is that your log contains "time" as time prefix and "time: " as the string really preceeding the actual timestamp. You could be more precise about that in case you have the word "time" somewhere within the event - you don't want splunk to have more work trying to guess where your timestamp is - the more precise you are, the better and more efficiently splunk works.

Another thing is that you don't have your timeformat defined. Don't leave splunk guessing whether something is a timestamp or not. It's best to have both TIME_PREFIX defined as precisely as possible and TIME_FORMAT set to the timestamp format you're expecting. It makes a huge difference on the input performance.

Next, you don't have MAX_TIMESTAMP_LOOKAHEAD set so it's probably at default 128 characters. Your events seem relatively long and the timestamp isn't placed at the beginning of the event so you might run into trouble if your "data part" of any particular event gets too long and timestamp gets pushed further down the event.

And finally - the timestamps look pretty much OK in terms of the value but shifted in timezones. There can be several reasons - the timezone from the timestamp could have been misinterpreted or you can have wrongly set timezone on your end (in the GUI). Remember that if you're located in a different timezone than reported in the event, the timestamp will get rendered by the GUI according to your timezone. Your events (at least one of them) is supposed to be in Zulu time (GMT). So if you're in another timezone, splunk's GUI may show it as a different hour.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Jackinout9 ,

as I supposed there are two times in the logs and sometimes Splunk doesn't take the correct one, you have to define TIME_PREFIX and TIME_FORMAT.

could you put the sample logs in the "Insers/Edit Code" Sample box?

I cannot find the regex to extract TIME_PREFIX and TIME_FORMAT.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...

What’s New in Splunk Observability – September 2025

What's NewWe are excited to announce the latest enhancements to Splunk Observability, designed to help ITOps ...

Fun with Regular Expression - multiples of nine

Fun with Regular Expression - multiples of nineThis challenge was first posted on Slack #regex channel ...