topic Re: Time is wrong. Events showing in the past (which are present) in Getting Data In

Time is wrong. Events showing in the past (which are present)

sirsam28 — Mon, 11 Dec 2023 18:44:22 GMT

Hi,

Rather new to splunk. I got some logs ingested but they are showing Time incorrectly. I have my TZ set on the UF server, splunk server and in my preferences as EST but I am getting this:

12/11/23
8:35:24.000 AM

2023-12-11T13:35:24+00:00 routerXXXXXX

If I look at the field _time I have: 2023-12-11T08:35:24.000-05:00

I suspect the source host or I need a props.conf to fix?

Re: Time is wrong. Events showing in the past (which are present)

PickleRick — Mon, 11 Dec 2023 19:03:58 GMT

And that seems about right. Your router reports 13:35GMT so Splunk parses it as 13:35GMT and shows it to you in your local time zone.

Your data quality is poor - configure your router to either report proper time zone or proper time (or even better - to report proper time in proper timezone).

Re: Time is wrong. Events showing in the past (which are present)

richgalloway — Mon, 11 Dec 2023 19:05:30 GMT

Please share the props.conf stanza for that sourcetype. It looks like the TIME_FORMAT string may be incorrect.

Re: Time is wrong. Events showing in the past (which are present)

sirsam28 — Mon, 11 Dec 2023 19:09:38 GMT

So this is a new install and new source.

In the splunk server there is no props.conf file. I assume I have to create it?

Re: Time is wrong. Events showing in the past (which are present)

richgalloway — Mon, 11 Dec 2023 20:05:45 GMT

Every sourcetype should have a stanza in props.conf. Create a props.conf file if there isn't a local copy already.

The stanza should contain these settings, at a minimum:

SHOULD_LINEMERGE LINE_BREAKER TIME_PREFIX TIME_FORMAT MAX_TIMESTAMP_LOOKAHEAD TRUNCATE EVENT_BREAKER_ENABLE EVENT_BREAKER

Re: Time is wrong. Events showing in the past (which are present)

PickleRick — Mon, 11 Dec 2023 20:22:05 GMT

Whiile Splunk can sometimes guess the proper settings for the sourcetype (and sometimes - as shown in this case - does it quite well), as @richgalloway mentioned - it's good to have the so-called "great eight" defined for each sourcetype to make it work consistently and efficiently.

Having said that - in this particular case your main issue is wrong time in your events!

You have your router's time set to a wrong value. Configure it properly. Whether it's reported as UTC or your local timezone is secondary as long as the proper timezone information is supplied (and in your case it is).

Re: Time is wrong. Events showing in the past (which are present)

sirsam28 — Mon, 11 Dec 2023 21:02:17 GMT

So interesting enough, in the router's cli:

xxxx@router:~ # date
Mon Dec 11 15:55:57 EST 2023

Also in the GUI showing correctly. I think a props.conf might be the route as it doesnt know how to translate it?

Would anyone be able to help craft an example stanza for it? I just dont want to mess up the logging further

Re: Time is wrong. Events showing in the past (which are present)

richgalloway — Mon, 11 Dec 2023 21:43:11 GMT

Give these settings a go

SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%dT%H:%M:%D%:z MAX_TIMESTAMP_LOOKAHEAD = 30 TRUNCATE = 10000 EVENT_BREAKER_ENABLE = true EVENT_BREAKER = ([\r\n]+)

Re: Time is wrong. Events showing in the past (which are present)

PickleRick — Mon, 11 Dec 2023 22:34:24 GMT

That is indeed interesting because supposedly keeping track of the timezome but in the end sending the timestamp with local time but explicitly saying that's UTC is not even a mistake. It's almost a crime. What ingenious piece of equipment is that if you can share this with us?

Re: Time is wrong. Events showing in the past (which are present)

sirsam28 — Tue, 12 Dec 2023 12:48:37 GMT

So its a opnsense firewall

Re: Time is wrong. Events showing in the past (which are present)

sirsam28 — Tue, 12 Dec 2023 13:01:18 GMT

Thanks for that. I created the file in /opt/splunk/etc/system/local/props.conf as follows:

[default] [host::router.xxxxxxxx] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%dT%H:%M:%D%:z MAX_TIMESTAMP_LOOKAHEAD = 30 TRUNCATE = 10000 EVENT_BREAKER_ENABLE = true

I am still getting the descrepency. Perhaps my props.conf file is not the correct format or in the right spot for Splunk to read?

Re: Time is wrong. Events showing in the past (which are present)

richgalloway — Tue, 12 Dec 2023 13:19:34 GMT

On which instance did you install those settings? They should be on the indexers (and heavy forwarders, if you have them). Did you restart the instances after modifying the file? Are you looking at new data? The changes will not affect indexed data. Do you have the correct host name in the stanza? Have you tried using the sourcetype name instead of host?