topic Re: Own timestamp assignment not working in Splunk Enterprise

Own timestamp assignment not working

outofheapspace — Tue, 29 Sep 2020 13:43:39 GMT

Hi,

I want to use timechart to analyse historical data files. The (*.log) files are just generated and each looks like this:

date = 1999-05-15
data1 = x
data2 = y

Now I want that "_time" contains my own "date" field. I found the article http://docs.splunk.com/Documentation/Splunk/6.5.3/Data/ConfigurePositionalTimestampExtraction
but I have a problem to bring this to work.

My environment: Forwarder (Windows 10), Indexer based on CentOS.
I cannot find any active props.conf.

I don't know if I need to place a props.conf in
/opt/splunk/etc/system/local (indexer) or
C:\Program Files\SplunkUniversalForwarder\etc\system\local (Forwarder)

I tried both and created a new props.conf with the following:

[host::FORWARDERHOSTNAME]
TIME_PREFIX = date =
TIME_FORMAT = %Y-%m-%d

And restarted all services. But the .log files I'm indexing will still be marked with a (_time) timestamp of import time (today) and not my historical date time (1999).

Some ideas?
Many thanks.

Re: Own timestamp assignment not working

gokadroid — Sat, 15 Apr 2017 05:49:58 GMT

How about trying this stanza to be put in your props.conf:

DATETIME_CONFIG = 
MAX_DAYS_AGO = 7300
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d
TIME_PREFIX = date =
category = Custom
pulldown_type = true

I am attaching the jpg where I used some sample data locally and used additional MAX_DAYS_AGO setting to make the dates in your sample data to parse correctly into _time.

MAX_DAYS_AGO = 7300 is actually 20 years worth of days ignoring leap years (20*365). So if you think your data might have dates prior to that, choose this value accordingly. Let me know if this helped.

Re: Own timestamp assignment not working

outofheapspace — Tue, 29 Sep 2020 13:43:50 GMT

Hi gokadroid,

it was not clear for me, that it's possible to configure that in the gui. I created a sourcetype and configured all above. After that I deleted my index and recreated a new one (Think it is possible to keep the old index!? - But that was a faster solution for me.). Now all Events are indexed with the correct date.
Also I can see the props.conf file now on the indexer (/opt/splunk/etc/apps/local) too.
WOW! That's exactly what I need. Now it is possible to use the timepicker with date range. First I tried it with reformatting the _time value ( | eval epoch = strptime(date, "%Y-%m-%d") | eval _time = epoch |) but that is not working with my time picker.
MAX_DAYS_AGO is also very usefull for me.
I thank you so much for that.