topic Re: Splunk time difference in Getting Data In

Why is Splunk showing a time difference?

Jackinout9 — Wed, 22 Mar 2023 22:02:30 GMT

Hi
We are trying to write the props from couple of days

Issue: splunk showing time difference 4 to 5 hours

logs are coming from one source with multiple time differences ..
example 1.
splunk time 3:48pm, log time 20:48 .
example 2.
splunk time 2:24pm log time 18:24.
time format : 2023-03-10T20:48:11.689534088Z

Please let me know if you have any ideas or solutions that could help us out here!

Re: Splunk time difference

gcusello — Tue, 21 Mar 2023 10:19:13 GMT

Hi @Jackinout9 ,

could you share a sample of events, both correctly and not correctly indexed?

maybe the problem is the presence of another time inside the events and the timestamp isn't correctle read.

Ciao.

Giuseppe

Re: Splunk time difference

Jackinout9 — Tue, 21 Mar 2023 10:34:29 GMT

Hi @gcusello

I tried with below props

[ <SOURCETYPE NAME> ]

SHOULD LINEMERGE=false

LINE BREAKER= ([\r\n]+)

NO BINARY CHECK=true

TIME PREFIX="time"

CHARSET=UTF-8

disabled=false

Sample logs :

3/10/23

3:48:11.689 PM

{ [-]

log: [20:48:11] [<unknown>][9f9835b5][ExtensionHostConnection] <417845> Extension Host Process exited with code: 0, signal: null.

stream: stdout

time: 2023-03-10T20:48:11.689534088Z

}

Show as raw text

3/13/23

2:24:20.526 PM

{ [-]

log: [18:24:20] [unknown][7fafc71d][ManagementConnection] New connection

established.

stream: stdout

time: 2023-03-13T18:24:20, 5264506632

}

Show as raw text

And also I attached sample pic

Re: Splunk time difference

gcusello — Tue, 21 Mar 2023 13:04:23 GMT

Hi @Jackinout9 ,

as I supposed there are two times in the logs and sometimes Splunk doesn't take the correct one, you have to define TIME_PREFIX and TIME_FORMAT.

could you put the sample logs in the "Insers/Edit Code" Sample box?

I cannot find the regex to extract TIME_PREFIX and TIME_FORMAT.

Ciao.

Giuseppe

Re: Splunk time difference

PickleRick — Tue, 21 Mar 2023 13:57:14 GMT

One thing is that your log contains "time" as time prefix and "time: " as the string really preceeding the actual timestamp. You could be more precise about that in case you have the word "time" somewhere within the event - you don't want splunk to have more work trying to guess where your timestamp is - the more precise you are, the better and more efficiently splunk works.

Another thing is that you don't have your timeformat defined. Don't leave splunk guessing whether something is a timestamp or not. It's best to have both TIME_PREFIX defined as precisely as possible and TIME_FORMAT set to the timestamp format you're expecting. It makes a huge difference on the input performance.

Next, you don't have MAX_TIMESTAMP_LOOKAHEAD set so it's probably at default 128 characters. Your events seem relatively long and the timestamp isn't placed at the beginning of the event so you might run into trouble if your "data part" of any particular event gets too long and timestamp gets pushed further down the event.

And finally - the timestamps look pretty much OK in terms of the value but shifted in timezones. There can be several reasons - the timezone from the timestamp could have been misinterpreted or you can have wrongly set timezone on your end (in the GUI). Remember that if you're located in a different timezone than reported in the event, the timestamp will get rendered by the GUI according to your timezone. Your events (at least one of them) is supposed to be in Zulu time (GMT). So if you're in another timezone, splunk's GUI may show it as a different hour.