Time is wrong. Events showing in the past (which are present)

[sirsam28]

Hi,

Rather new to splunk. I got some logs ingested but they are showing Time incorrectly. I have my TZ set on the UF server, splunk server and in my preferences as EST but I am getting this:

12/11/23 8:35:24.000 AM

2023-12-11T13:35:24+00:00 routerXXXXXX

If I look at the field _time I have: 2023-12-11T08:35:24.000-05:00

I suspect the source host or I need a props.conf to fix?

[richgalloway]

Please share the props.conf stanza for that sourcetype.  It looks like the TIME_FORMAT string may be incorrect.

[sirsam28]

So this is a new install and new source. 

In the splunk server there is no props.conf file. I assume I have to create it?

[PickleRick]

Whiile Splunk can sometimes guess the proper settings for the sourcetype (and sometimes - as shown in this case - does it quite well), as @richgalloway mentioned - it's good to have the so-called "great eight" defined for each sourcetype to make it work consistently and efficiently.

Having said that - in this particular case your main issue is wrong time in your events!

You have your router's time set to a wrong value. Configure it properly. Whether it's reported as UTC or your local timezone is secondary as long as the proper timezone information is supplied (and in your case it is).

[sirsam28]

So interesting enough, in the router's cli:

xxxx@router:~ # date
Mon Dec 11 15:55:57 EST 2023

Also in the GUI showing correctly. I think a props.conf might be the route as it doesnt know how to translate it?

Would anyone be able to help craft an example stanza for it? I just dont want to mess up the logging further

[PickleRick]

That is indeed interesting because supposedly keeping track of the timezome but in the end sending the timestamp with local time but explicitly saying that's UTC is not even a mistake. It's almost a crime. What ingenious piece of equipment is that if you can share this with us?

[sirsam28]

So its a opnsense firewall 

[richgalloway]

Give these settings a go

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%D%:z
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE = 10000
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)

[sirsam28]

Thanks for that. I created the file in /opt/splunk/etc/system/local/props.conf as follows:

[default]


[host::router.xxxxxxxx]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%D%:z
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE = 10000
EVENT_BREAKER_ENABLE = true

I am still getting the descrepency. Perhaps my props.conf file is not the correct format or in the right spot for Splunk to read?

[richgalloway]

On which instance did you install those settings?  They should be on the indexers (and heavy forwarders, if you have them).  Did you restart the instances after modifying the file?  Are you looking at new data?  The changes will not affect indexed data.  Do you have the correct host name in the stanza?  Have you tried using the sourcetype name instead of host?

[richgalloway]

Every sourcetype should have a stanza in props.conf.  Create a props.conf file if there isn't a local copy already.

The stanza should contain these settings, at a minimum:

SHOULD_LINEMERGE
LINE_BREAKER
TIME_PREFIX
TIME_FORMAT
MAX_TIMESTAMP_LOOKAHEAD
TRUNCATE
EVENT_BREAKER_ENABLE
EVENT_BREAKER

[PickleRick]

And that seems about right. Your router reports 13:35GMT so Splunk parses it as 13:35GMT and shows it to you in your local time zone.

Your data quality is poor - configure your router to either report proper time zone or proper time (or even better - to report proper time in proper timezone).