_time is being set to mtime instead of parsed event time

[punkle64]

I have the following source log files:

[root@lts-reporting ~]# head /nfs/LTS/splunk/lts12_summary.log
2014-07-01T00:00:00 78613376660548
2014-08-01T00:00:00 94340587484234
2014-09-01T00:00:00 105151971182496
2014-10-01T00:00:00 104328846250489
2014-11-01T00:00:00 124100293157039
2014-12-01T00:00:00 150823795700989
2015-01-01T00:00:00 178786111756322
2015-02-01T00:00:00 225445840948631
2015-03-01T00:00:00 248963904047438
2015-04-01T00:00:00 274070504403562

[root@lts-reporting ~]# head /nfs/LTS/splunk/lts22_summary.log
2014-07-01T00:00:00 87011545030617
2014-08-01T00:00:00 112491174858354
2014-09-01T00:00:00 114655842870462
2014-10-01T00:00:00 102729950441541
2014-11-01T00:00:00 124021498471043
2014-12-01T00:00:00 147319995334181
2015-01-01T00:00:00 182983059554298
2015-02-01T00:00:00 234679634668451
2015-03-01T00:00:00 252420788862798
2015-04-01T00:00:00 288156185998535

On the universal forwarder I have the following inputs.conf stanzas:

## LTS summaries
[monitor:///nfs/LTS/splunk/lts12_summary.log]
_TCP_ROUTING = druid
index = lts
sourcetype = size_summaries

## LTS summaries
[monitor:///nfs/LTS/splunk/lts22_summary.log]
_TCP_ROUTING = druid
index = lts
sourcetype = size_summaries

I have the following splunk props stanza:

## LTS Size Summaries
[size_summaries]
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG = NONE
EXTRACT-ltsserver = /nfs/LTS/splunk/(?<ltsserver>\w+)_summary.log in source
EXTRACT-size = (?m)^\S+\s+(?<size>\d+)

When indexing the files for the first time the events get parsed with the correct _time (the first field in every line of the log), but when a new event gets logged all the events get assigned the latest modification time of the log file. I have tried deleting the events by sourcetype on the indexer and restarted splunk to see if anything changes, but I get exactly the same behaviour. Unfortunately I cannot delete the full fishbucket of the index as I have other source types in the same index which would be lost.

Is there a way to force the indexer to parse the first field of the events as _time?

[punkle64]

Ok I quickly managed to reindex the file through a temporary crcSalt in the forwarder inputs.conf. The events are now indexed fine thanks to removing the

DATETIME_CONFIG=NONE 

Many thanks to PickleRick for providing the solution.

[livehybrid]

Hi @punkle64 

Please can you confirm that your props.conf is on your HF or Indexer - not the UF? The index time parsing will be done on the first "full" instance if Splunk if reaches (Heavy Forwarder / Indexer). 

The other thing you might need to check is increasing the MAX_DAYS_AGO value - it could be that the date detected is too far away and Splunk is defaulting to the modified time.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[punkle64]

Many thanks for your answer. Yes, the props.conf is on the indexer and works fine for the other fields (ltsserver and size) so it means the config is taken into account. The only thing which doesn't work is the _time. I tried as you suggested to increase the MAX_DAYS_AGO to the maximum (10951), but I am always getting the same results:

index=lts sourcetype=size_summaries | table _time _raw

2025-04-23 14:47:13	2019-02-01T00:00:00 3830390070938120
2025-04-23 14:47:13	2019-01-01T00:00:00 3682803389795110
2025-04-23 14:47:13	2018-12-01T00:00:00 3583659674663620
2025-04-23 14:47:13	2018-11-01T00:00:00 3500420740998170
2025-04-23 14:47:13	2018-10-01T00:00:00 3439269816258840
2025-04-23 14:47:13	2018-09-01T00:00:00 3365435411968590

[PickleRick]

Check your _internal for warnings. Typically when the events are that old, you need to tweak the age/difference related settings (MAX_DAYS_AGO is one of them). Otherwise Splunk decides that it must have parsed the time wrongly because the timestamp doesn't make sense and - depending on situation - assumes current timestamp or copies over the last one.

[punkle64]

I have added even more settings to the props.conf:

MAX_DAYS_AGO = 10951
MAX_DAYS_HENCE = 10950
MAX_DIFF_SECS_AGO = 2147483646
MAX_DIFF_SECS_HENCE = 2147483646

and checked the _internal but there are no warnings. Unfortunately there are no improvements:

2025-04-23 17:06:05	2023-12-01T00:00:00 11557603686635900
2025-04-23 17:06:05	2023-11-01T00:00:00 11341507392715400
2025-04-23 17:06:05	2023-10-01T00:00:00 11116993118051800
2025-04-23 17:06:05	2023-09-01T00:00:00 10521042084168300
2025-04-23 17:06:05	2023-08-01T00:00:00 10017490857052000
2025-04-23 17:06:05	2023-07-01T00:00:00 9691291660267240

Isn't there a workaround to force _time to take its value from a specific field?

[PickleRick]

Wait a second. Earlier you wrote (which I had missed) that your other extracted fields do work. Since you're using search-time extractions, that means that you have props defined on your search head tier. Do you also have those props on the correct component in ingestion path? What is your architecture and where do you have which settings?

[punkle64]

Hello PickleRick,

The architecture is simple: I have UniversalForwarders on around 30 servers with /opt/splunkforwarder/etc/apps/druid_forwarder/default/inputs.conf (contents is in the first post) and then I have 1 indexer with /opt/splunk/etc/apps/druid_utils/default/props.conf (contents is in the first post). The inputs.conf is only on the universal forwarder(s) while the props.conf is only on the indexer.

[PickleRick · Accepted solution]

Ah, OK. So you have an all-in-one instance, not a standalone indexer.

In that case indeed props should be on the AIO box.

But.

You have

DATETIME_CONFIG=NONE

Quoting spec:

* This setting may also be set to "NONE" to prevent the timestamp
  extractor from running or "CURRENT" to assign the current system time to
  each event.

Just remove this line from your config.

[punkle64]

Ok, I removed the:

DATETIME_CONFIG=NONE

But now all events are gone. I also tried to empty the index:

/opt/splunk/bin/splunk clean eventdata -index lts
This action will permanently erase all events from the index 'lts'; it cannot be undone.
Are you sure you want to continue [y/n]? y
Cleaning database lts.

And restarted both the indexer and the forwarder. But no events are sent.

[PickleRick]

The DATETIME_CONFIG setting has nothing to do with already indexed data so something else must have happened if you "lost" your events.

[punkle64]

Hello PickleRick,

Apologies: I had the wrong impression. The events were not really gone: I deleted them myself, hoping that they would be reindexed immediately with the correct _time, as they did in the previous trials. This time however, after deleting them from the indexer they were not reindexed automatically, so I incorrectly used the word "gone". So I added a crcSalt temporarily in the inputs.conf of the forwarder, all events were reindexed with the correct _time and everything worked perfectly thanks to your suggestion.

Many thanks again for your solution 🙂