topic Re: Importing 'old' data set results in incorrect date extraction after 2010 in Getting Data In

Importing 'old' data set results in incorrect date extraction after 2010

cam343 — Wed, 25 Nov 2015 10:43:01 GMT

Hello,
Trying to import a CSV with dates going back 50+ years (https://www.quandl.com/api/v3/datasets/BCB/UDJIAD1.csv).
I have correctly set the props date extraction and it works, except for events after 2010.
What's happened is all the "old" dates are now bunched up at 6/7/2010 and haven't extracted properly.

I know their is quarantine settings for indexes.conf but that should still display the data correctly it just puts it into different buckets on disk.....

Is there a limitation of how 'old' data you can import? How can I fix it?

[ dow_jones_index_csv ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIMESTAMP_FIELDS=Date
TIME_FORMAT=%Y-%m-%d
category=Structured
description=Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled=false
pulldown_type=true

Re: Importing 'old' data set results in incorrect date extraction after 2010

cam343 — Wed, 25 Nov 2015 11:35:25 GMT

I tried increasing to:
frozenTimePeriodInSecs = 4294967295

but it didnt make a difference....

Re: Importing 'old' data set results in incorrect date extraction after 2010

woodcock — Wed, 25 Nov 2015 14:16:05 GMT

You'll likely need to increase MAX_DAYS_AGO in props.conf:

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, that an extracted date  can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.