Hello,
Trying to import a CSV with dates going back 50+ years (https://www.quandl.com/api/v3/datasets/BCB/UDJIAD1.csv).
I have correctly set the props date extraction and it works, except for events after 2010.
What's happened is all the "old" dates are now bunched up at 6/7/2010 and haven't extracted properly.
I know their is quarantine settings for indexes.conf but that should still display the data correctly it just puts it into different buckets on disk.....
Is there a limitation of how 'old' data you can import? How can I fix it?
[ dow_jones_index_csv ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIMESTAMP_FIELDS=Date
TIME_FORMAT=%Y-%m-%d
category=Structured
description=Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled=false
pulldown_type=true
You'll likely need to increase MAX_DAYS_AGO
in props.conf
:
MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.
I tried increasing to:
frozenTimePeriodInSecs = 4294967295
but it didnt make a difference....