Events in the Future from a UF

[b17gunnr]

Hello folks,

I have a universal forwarder sending Splunk logs which follow the format below. Unfortunately Splunk is not recognizing the UTC offset and is declaring them as being the future. I have edited my /opt/splunk/etc/system/local/props.conf file with the following stanza and restarted but there is no change and Splunk still wants them in UTC. Can anyone point me at what I'm doing wrong? Thank you.

Log Entry
2025-10-29T14:55:17-04:00 cis-pfmc SF-IMS[5970]: [meta sequenceId="5415975"]CSMEventServer[11588]: *** Got a message:

------------------------------------

Proposed /opt/splunk/etc/system/local/props.conf stanza
[audit:log]
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
TZ = UTC

------------------------------------

Additional Info
index=firewall_audit
sourcetype=audit:log
source=/var/log/firewall.log

[shashankD]

Rather then using the time zone stanza in props.conf, use timezone stanza in inputs.conf, this will resolve the future time stamp issue. I was facing the same issue with my multi site UF (specificially APAC region - Japan, australia, india) and this helped me.

[PickleRick]

With timestamp assignment set to CURRENT the TZ setting doesn't matter since the timestamp is getting assigned based on the receving component's OS time.

[b17gunnr]

The problem has been resolved with the following stanza in /opt/splunk/etc/system/local/props.conf

[audit:log]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%:z
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
# TZ = UTC
MAX_DIFF_SECS_HENCE = 86400
DATETIME_CONFIG = CURRENT

Some additional research. Prior to adding the bottom two lines to the stanza, I verified the VM is aware of the correct date and time, that it was distinguishing between local and UTC properly (note that the change from EDT to EST moved the offset from 4 to 5 hours), and that we were sync'd to NTP.

splunk@idx1:~$ date
Tue Nov 4 11:32:35 AM EST 2025

splunk@idx1:~$ timedatectl
Local time: Tue 2025-11-04 11:32:47 EST
Universal time: Tue 2025-11-04 16:32:47 UTC
RTC time: Tue 2025-11-04 16:32:47
Time zone: America/New_York (EST, -0500)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no

splunk@idx1:~$ timedatectl show | grep NTPSynchronized
NTPSynchronized=yes

Then verified that there was nothing abnormal on the search heads which could be taking priority of the changes made to the indexer.

splunk@srch01:~$ /opt/splunk/bin/splunk btool props list audit:log --debug
splunk@srch01:~$
splunk@srch02:~$ /opt/splunk/bin/splunk btool props list audit:log --debug
splunk@srch03:~@
splunk@srch03:~$ /opt/splunk/bin/splunk btool props list audit:log --debug
splunk@srch03:~$

[PickleRick]

You effectively broke the event recognition completely. DATETIME_CONFIG = CURRENT means that you're not recognizing the timestamp from the event but assuming the current timestamp of the receiving component. That is wrong. If you have any delays in the ingestion process, your data will not be properly timestamped.

This is _not_ the solution of the original problem. This is a way of a short-term avoidance of the problem but not the solution.

[livehybrid]

HI @b17gunnr 

Please can you confirm where you are making these props.conf changes? 

They need to be on the first HF (or Indexer) that the data hits, not the UF. 

Additionally, you shouldnt need to set the TZ=UTC because you have the timezone extraction in the  TIME_FORMAT setting.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[b17gunnr]

Hello,

I'm working on the indexer's props.conf. I've tried it with and without TZ = UTC and it's still reading 4 hours into the future.

[PickleRick]

What do you mean by "4 hours into the future". The timestamp you posted in the example event references 18:55:17 UTC (1761764117 as epoch-based unix timestamp).

[b17gunnr]

The indexer is not recognizing the then 4 hour offset from UTC; I say then because we flipped form EDT to EST over the weekend. So, when an event is received by the indexer at 08:00:00 EDT, the 12:00:00UTC time is being recognized and Splunk holds that event until noon EDT.

[richgalloway]

I see that happen a LOT.  It's usually because the TIME_FORMAT string does not account for the time zone in the event, but you say you've done that.  Here are some things to check

• props.conf settings are correct, including attribute spelling and regular expressions
• Props are installed on the right instance(s)
• Props are not overridden by settings in other apps.  Use btool to check that.
• Instances are restarted after making props changes.
• Props in sourcetype stanzas are not overridden by props in host:: or source:: stanzas

[PickleRick]

Are you sure it's not something on your end? With the time(zone) configuration on your SH?

What do you get if you run

| makeresults 
| eval _time=strptime("2025-10-29T14:55:17-04:00","%Y-%m-%dT%H:%M:%S%z")
| eval time=strftime(_time,"%s")

[richgalloway]

The TIME_FORMAT setting doesn't match the data.  Try

TIME_FORMAT = %Y-%m-%dT%H:%M:%S%:z

[PickleRick]

No. It actually does match the format. It's a bit unfortunate that the offset is "glued" to the time part but it seems to match.

And since the timestamp references timezone definition, the TZ setting for the sourcetype should be ignored.

So there's something else here. Are the settings defined in the right place on the right component? (and aren't they overwritten by settings from another app?)

[b17gunnr]

I believe I have the stanza in the correct location, here's a btool output and the sourcetype in question is only in one of two places.

splunk@splk-indexer01:~$ /opt/splunk/bin/splunk btool props list audit:log --debug
/opt/splunk/etc/system/local/props.conf   [audit:log]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/system/default/props.conf HEADER_MODE = 
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_EXPECTED_EVENT_LINES = 7
/opt/splunk/etc/system/local/props.conf   MAX_TIMESTAMP_LOOKAHEAD = 30
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = True
/opt/splunk/etc/system/local/props.conf   TIME_FORMAT = %Y-%m-%dT%H:%M:%S%:z
/opt/splunk/etc/system/local/props.conf   TIME_PREFIX = ^
/opt/splunk/etc/system/default/props.conf TRANSFORMS = 
/opt/splunk/etc/system/local/props.conf   TRUNCATE = 75000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority = 
/opt/splunk/etc/system/default/props.conf sourcetype = 
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false
/opt/splunk/etc/system/default/props.conf unarchive_cmd_start_mode = shell

[b17gunnr]

Hello,

I've tried the suggested TIME-FORMAT with and without TZ = UTC and it's still reading 4 hours into the future.

[richgalloway]

The TIME_FORMAT setting must be on the first full Splunk instance that receives the data.  That's usually an indexer, but could be a heavy forwarder.  The TZ should be on the UF.  Both instance types must be restarted after changing a props.conf file before the change will take effect.

Changes to TIME_FORMAT, TZ, and other props settings will not change events that are already indexed.  They only affect new data as it arrives.

[b17gunnr]

Struck out here as well.

[ddrillic]

@b17gunnr do you mind describing the architecture? per the direction of @richgalloway  

[b17gunnr]

Hello,

It is a straight-forward solution. I have a UF that is reading an internal file and forwarding contents to the Indexer. The Indexer is receiving those updates in UTC minus US eastern time offset (now 5 hours) but is not adjusting as per changes made to the Indexer's props.conf. All suggestions were tried and restarts took place after each change. No change in status...