Getting Data In

Importing 'old' data set results in incorrect date extraction after 2010

cam343
Path Finder

Hello,
Trying to import a CSV with dates going back 50+ years (https://www.quandl.com/api/v3/datasets/BCB/UDJIAD1.csv).
I have correctly set the props date extraction and it works, except for events after 2010.
What's happened is all the "old" dates are now bunched up at 6/7/2010 and haven't extracted properly.

I know their is quarantine settings for indexes.conf but that should still display the data correctly it just puts it into different buckets on disk.....

Is there a limitation of how 'old' data you can import? How can I fix it?

[ dow_jones_index_csv ]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIMESTAMP_FIELDS=Date
TIME_FORMAT=%Y-%m-%d
category=Structured
description=Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled=false
pulldown_type=true
0 Karma

woodcock
Esteemed Legend

You'll likely need to increase MAX_DAYS_AGO in props.conf:

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, that an extracted date  can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

cam343
Path Finder

I tried increasing to:
frozenTimePeriodInSecs = 4294967295

but it didnt make a difference....

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...