topic Re: handling future dates from csv in Getting Data In

handling future dates from csv

dsiob — Tue, 13 Jun 2017 11:43:38 GMT

hi,
I have a csv file, having time field 'Start Time'. This field will have entries of current week dates as well as future week dates. Eg: If today is 6/13/2017, It will have 6/16/2017, 6/23/2017 etc. When I try to upload the csv it shows warning 'Could not use regex to parse timestamp '6/23/2017'. And when start searching on the csv, it shows 'NO result found'.
I have to have future dates in the csv.
Can somebody help to overcome this problem?

Re: handling future dates from csv

woodcock — Tue, 13 Jun 2017 11:56:02 GMT

The root cause of this problem is that the limit MAX_DAYS_HENCE (which is 2 by default) is deliberately disallowing your events.
It is very easy to increase MAX_DAYS_HENCE in props.conf to tolerate your amazing events from the future.

To double-check if this is the case, you should also be seeing warning logs like this:

10-12-2010 12:53:32.619 WARN DateParserVerbose - The TIME_FORMAT specified is matching timestamps (Tue Oct 12 12:53:32 2010) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.

You can run this search to see:

index="internal" sourcetype="splunkd" DateParserVerbose MAX_DAYS* "outside of the acceptable time window"
| rex "matching timestamps ((?<ts>[^)]+))"
| eval ts2=strptime(ts, "%a %b %d %T %Y")
| eval time_diff=round((_time-ts2)/60,1)
| where abs(time_diff)<=1440
| table _time, ts, time_diff, host

You will have to restart all splunk instances on your Indexers after you change this setting.

Re: handling future dates from csv

dsiob — Tue, 29 Sep 2020 14:26:51 GMT

I changed in file C:\Program Files\Splunk\etc\system\default\props.config, MAX_DAYS_HENCE=30. But it is not reflecting.

Re: handling future dates from csv

woodcock — Tue, 13 Jun 2017 12:11:59 GMT

Did you deploy to your INDEXERS and restart all splunk instances there? This will only apply to events you index AFTER the change (older broken events will "stay missing").

Re: handling future dates from csv

dsiob — Tue, 13 Jun 2017 12:27:11 GMT

'Did you deploy to your INDEXERS' can you pls tell me how to do that.

Re: handling future dates from csv

dsiob — Tue, 13 Jun 2017 12:32:15 GMT

i am uploading csv and using it without specify any index

Re: handling future dates from csv

woodcock — Tue, 13 Jun 2017 12:36:27 GMT

You should change that back, create a C:\Program Files\Splunk etc\system\local\props.conf file and put in it this:

[default]
MAX_DAYS_HENCE=300

Then, assuming that you are using a splunk all-in-one instance on your personal computer, go to Settings -> Server controls -> Restart.

Re: handling future dates from csv

dsiob — Tue, 13 Jun 2017 13:20:43 GMT

thanks Woodcock..its working!