My Splunkcloud _time is 4 hours in the future

[marcluescher]

Hey Splunkers,

while I was able to be self sufficient in most cases I have one application log server which is driving me crazy. We have working filemonitor but even all files are in the correct timezone on the sourcesystem the end up 4 hours in the future on Splunk cloud. All other log files from the same middleware application park are correct. We deployed to our UF and  allHF's before sending to Splunk cloud

Our props.conf is :

[oid.prod:log]
MAX_TIMESTAMP_LOOKAHEAD = 26
TZ = UTC
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%6N
LINE_BREAKER = ([\r\n]+)

source event is like :

2021-05-07 20:26:19.3921 "OID" - - - "83270" - - "UserLogout" TRUE - "cn=remedyadmin,cn=service_accounts,dc=dsd,dc=xxxxxx,dc=com" "Operation name: unbind" - "10.3.79.10" - - - - "unbind" -

the event time is:

2021-05-07T20:26:19.392-04:00

So how can I fix this so I can avoid to select All Time to display that sourcetype which is 4h in the future, all timezones are set to Easter in Splunk. I am at a loss here and have a similar issue with a syslog source on a different cloud based system.

[isoutamo]

You said in props.conf that source system time is in UTC/GMT time, but probably those are in your local TZ. Can you remove a TZ definition there or update it to point the correct TZ?

[richgalloway]

The TIME_FORMAT setting is incorrect for the different sample event times shown so Splunk is using the default and getting it wrong.  Also, the TZ=UTC setting will tend to put events 4 hours ahead of EDT.  The TZ setting should be on the UF and the other settings on your HFs and indexers.

[marcluescher]

So, let me see if i got it right :

on UF , you define the inputs.conf (filemonitor) and just the TZ in props.conf

on the HF, you define just the props.conf with TZ, time_format and time_prefix

what do I need on the Splunkcloud indexers and do need to create app to get it deployed there ?

Sorry for asking.

Marc

[richgalloway · Accepted solution]

Correct on the UF.

You do not need TZ on HFs, especially if it is receiving events from multiple time zones.

Since the HF is doing the event processing, you don't need any settings on the indexers.  Assuming, of course, that all data passes through an HF.  If anything is sent directly to the indexers then props.conf just be there as well.  I like to put props.conf there just in case.