Getting Data In

Splunk can't handle old timestamps

msarro
Builder

Greetings everyone. I am receiving a gamut of old files, some of which contain test data showing records from 1970. Splunk is indexing them using the file's modtime and not the timestamp, which is quickly diluting real data. Why doesn't splunk index them using the correct timestamp as shown in the file?

Tags (1)
0 Karma
1 Solution

Ayn
Legend

You'll likely want to increase MAX_DAYS_AGO in props.conf.

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, that an extracted date  can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

View solution in original post

msarro
Builder

Please don't be offended, but the fix mentioned below appears to be what is needed. The CDR is more than 300 fields long, and would take a LONG time to manually sanitize. The benefit would be limited since the below excerpt indicates that splunk only accepts timestamps in the past 2000 days - and the data we're seeing is a limited portion of the total sample.

0 Karma

araitz
Splunk Employee
Splunk Employee

Just one sanitized log sample would suffice. It is very hard to conjecture our way to solving your problem.

0 Karma

msarro
Builder

I would love to, however this is CDR data, and under federal regulations I can't share it samples without sanitizing them. However I am using 4.3. I am using a configured sourcetype, as well as 4 transforms. Although I believe the next poster's solution is the one I need.

0 Karma

Ayn
Legend

You'll likely want to increase MAX_DAYS_AGO in props.conf.

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, that an extracted date  can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

View solution in original post

bwooden
Splunk Employee
Splunk Employee

It would be helpful to those wanting to help if you would provide the version of Splunk you're using, a few sample lines of source data, and whether you are using auto sourcetyping or have configured your own sourcetype.

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!