Getting Data In

DATETIME_CONFIG = NONE doesn't work

hongduan
Explorer

I have this stanza in my props.conf
[test_test]
TZ = US/Eastern
SHOULD_LINEMERGE = false
MAX_DAYS_HENCE = 5
TRUNCATE = 0
DATETIME_CONFIG = NONE

And this setting in inputs.conf
[monitor:///test_folder/test.txt]
disabled = 0
index = test_index
sourcetype = test_test
crcSalt =

why splunk is still trying to parse the timestamp in each line of test.txt instead of using the modified time of test.txt?

Tags (1)

bnorthway
Path Finder

Unfortunately, the docs say:

"NONE" will leave the event time set to whatever time was selected by the input layer
For data sent by splunk forwarders over the splunk protocol, the input layer will be the time that was selected on the forwarder by its input behavior (as below).

My solution was to use a batch input:

[batch://D:\data*.csv]
sourcetype = foo
move_policy = sinkhole

and the sourcetype stanza in props.conf has DATETIME_CONFIG = NONE.

NOTE batch input will DELETE files as it imports them! So make a copy first!

0 Karma

starcher
Influencer

Are you running a single instance of splunk? Or distributed with dedicated indexers? Timestamp manipulations are an index time thing so it needs to be on the indexers.

0 Karma

hongduan
Explorer

when test this configuration in splunk data preview, it works very well. all records are using the modification time on the file
but after I add my file into splunk using that sourcetype, splunk tries to parse the timestamp in the log record again.

0 Karma

hongduan
Explorer

it is not on the indexers. it is on the sourcetype. Why cannot I have different configurations for each sourcetype in an indexer?

0 Karma

starcher
Influencer

You will need to put the props stanza on each of the indexers.

0 Karma

linu1988
Champion

is it stored in the indexers in your distributed environment?

0 Karma

hongduan
Explorer

distributed with decicated indexers

0 Karma

hongduan
Explorer

Does that matter, thought splunk will ignore those fields. Let me try remove those fields see if it works.

0 Karma

linu1988
Champion

if you don't need the date time config why is the TZ and Max_days_hence are there in your config?

0 Karma

hongduan
Explorer

I am sure the props.conf file is placed in the correct location. cause other changes I made to props.conf are picked up by splunk, only this DATATIME_CONFIG = NONE not work.

0 Karma

linu1988
Champion

where is you props.conf file placed?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...