Splunk Enterprise

Own timestamp assignment not working

outofheapspace
Explorer

Hi,

I want to use timechart to analyse historical data files. The (*.log) files are just generated and each looks like this:

date = 1999-05-15
data1 = x
data2 = y

Now I want that "_time" contains my own "date" field. I found the article http://docs.splunk.com/Documentation/Splunk/6.5.3/Data/ConfigurePositionalTimestampExtraction
but I have a problem to bring this to work.

My environment: Forwarder (Windows 10), Indexer based on CentOS.
I cannot find any active props.conf.

I don't know if I need to place a props.conf in
/opt/splunk/etc/system/local (indexer) or
C:\Program Files\SplunkUniversalForwarder\etc\system\local (Forwarder)

I tried both and created a new props.conf with the following:

[host::FORWARDERHOSTNAME]
TIME_PREFIX = date =
TIME_FORMAT = %Y-%m-%d

And restarted all services. But the .log files I'm indexing will still be marked with a (_time) timestamp of import time (today) and not my historical date time (1999).

Some ideas?
Many thanks.

Tags (1)
0 Karma
1 Solution

gokadroid
Motivator

How about trying this stanza to be put in your props.conf:

DATETIME_CONFIG = 
MAX_DAYS_AGO = 7300
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d
TIME_PREFIX = date =
category = Custom
pulldown_type = true

I am attaching the jpg where I used some sample data locally and used additional MAX_DAYS_AGO setting to make the dates in your sample data to parse correctly into _time.

MAX_DAYS_AGO = 7300 is actually 20 years worth of days ignoring leap years (20*365). So if you think your data might have dates prior to that, choose this value accordingly. Let me know if this helped.

alt text

View solution in original post

0 Karma

gokadroid
Motivator

How about trying this stanza to be put in your props.conf:

DATETIME_CONFIG = 
MAX_DAYS_AGO = 7300
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d
TIME_PREFIX = date =
category = Custom
pulldown_type = true

I am attaching the jpg where I used some sample data locally and used additional MAX_DAYS_AGO setting to make the dates in your sample data to parse correctly into _time.

MAX_DAYS_AGO = 7300 is actually 20 years worth of days ignoring leap years (20*365). So if you think your data might have dates prior to that, choose this value accordingly. Let me know if this helped.

alt text

0 Karma

outofheapspace
Explorer

Hi gokadroid,

it was not clear for me, that it's possible to configure that in the gui. I created a sourcetype and configured all above. After that I deleted my index and recreated a new one (Think it is possible to keep the old index!? - But that was a faster solution for me.). Now all Events are indexed with the correct date.
Also I can see the props.conf file now on the indexer (/opt/splunk/etc/apps/local) too.
WOW! That's exactly what I need. Now it is possible to use the timepicker with date range. First I tried it with reformatting the _time value ( | eval epoch = strptime(date, "%Y-%m-%d") | eval _time = epoch |) but that is not working with my time picker.
MAX_DAYS_AGO is also very usefull for me.
I thank you so much for that.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...