Getting Data In

DATETIME_CONFIG = NONE doesn't work

hongduan
Explorer

I have this stanza in my props.conf
[test_test]
TZ = US/Eastern
SHOULD_LINEMERGE = false
MAX_DAYS_HENCE = 5
TRUNCATE = 0
DATETIME_CONFIG = NONE

And this setting in inputs.conf
[monitor:///test_folder/test.txt]
disabled = 0
index = test_index
sourcetype = test_test
crcSalt =

why splunk is still trying to parse the timestamp in each line of test.txt instead of using the modified time of test.txt?

Tags (1)

bnorthway
Path Finder

Unfortunately, the docs say:

"NONE" will leave the event time set to whatever time was selected by the input layer
For data sent by splunk forwarders over the splunk protocol, the input layer will be the time that was selected on the forwarder by its input behavior (as below).

My solution was to use a batch input:

[batch://D:\data*.csv]
sourcetype = foo
move_policy = sinkhole

and the sourcetype stanza in props.conf has DATETIME_CONFIG = NONE.

NOTE batch input will DELETE files as it imports them! So make a copy first!

0 Karma

starcher
SplunkTrust
SplunkTrust

Are you running a single instance of splunk? Or distributed with dedicated indexers? Timestamp manipulations are an index time thing so it needs to be on the indexers.

0 Karma

hongduan
Explorer

when test this configuration in splunk data preview, it works very well. all records are using the modification time on the file
but after I add my file into splunk using that sourcetype, splunk tries to parse the timestamp in the log record again.

0 Karma

hongduan
Explorer

it is not on the indexers. it is on the sourcetype. Why cannot I have different configurations for each sourcetype in an indexer?

0 Karma

starcher
SplunkTrust
SplunkTrust

You will need to put the props stanza on each of the indexers.

0 Karma

linu1988
Champion

is it stored in the indexers in your distributed environment?

0 Karma

hongduan
Explorer

distributed with decicated indexers

0 Karma

hongduan
Explorer

Does that matter, thought splunk will ignore those fields. Let me try remove those fields see if it works.

0 Karma

linu1988
Champion

if you don't need the date time config why is the TZ and Max_days_hence are there in your config?

0 Karma

hongduan
Explorer

I am sure the props.conf file is placed in the correct location. cause other changes I made to props.conf are picked up by splunk, only this DATATIME_CONFIG = NONE not work.

0 Karma

linu1988
Champion

where is you props.conf file placed?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...