Getting Data In

handling future dates from csv

dsiob
Communicator

hi,
I have a csv file, having time field 'Start Time'. This field will have entries of current week dates as well as future week dates. Eg: If today is 6/13/2017, It will have 6/16/2017, 6/23/2017 etc. When I try to upload the csv it shows warning 'Could not use regex to parse timestamp '6/23/2017'. And when start searching on the csv, it shows 'NO result found'.
I have to have future dates in the csv.
Can somebody help to overcome this problem?

alt text

0 Karma
1 Solution

woodcock
Esteemed Legend

The root cause of this problem is that the limit MAX_DAYS_HENCE (which is 2 by default) is deliberately disallowing your events.
It is very easy to increase MAX_DAYS_HENCE in props.conf to tolerate your amazing events from the future.

To double-check if this is the case, you should also be seeing warning logs like this:

10-12-2010 12:53:32.619 WARN DateParserVerbose - The TIME_FORMAT specified is matching timestamps (Tue Oct 12 12:53:32 2010) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.

You can run this search to see:

index="internal" sourcetype="splunkd" DateParserVerbose MAX_DAYS* "outside of the acceptable time window"
| rex "matching timestamps ((?<ts>[^)]+))"
| eval ts2=strptime(ts, "%a %b %d %T %Y")
| eval time_diff=round((_time-ts2)/60,1)
| where abs(time_diff)<=1440
| table _time, ts, time_diff, host

You will have to restart all splunk instances on your Indexers after you change this setting.

View solution in original post

0 Karma

woodcock
Esteemed Legend

The root cause of this problem is that the limit MAX_DAYS_HENCE (which is 2 by default) is deliberately disallowing your events.
It is very easy to increase MAX_DAYS_HENCE in props.conf to tolerate your amazing events from the future.

To double-check if this is the case, you should also be seeing warning logs like this:

10-12-2010 12:53:32.619 WARN DateParserVerbose - The TIME_FORMAT specified is matching timestamps (Tue Oct 12 12:53:32 2010) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.

You can run this search to see:

index="internal" sourcetype="splunkd" DateParserVerbose MAX_DAYS* "outside of the acceptable time window"
| rex "matching timestamps ((?<ts>[^)]+))"
| eval ts2=strptime(ts, "%a %b %d %T %Y")
| eval time_diff=round((_time-ts2)/60,1)
| where abs(time_diff)<=1440
| table _time, ts, time_diff, host

You will have to restart all splunk instances on your Indexers after you change this setting.

0 Karma

dsiob
Communicator

I changed in file C:\Program Files\Splunk\etc\system\default\props.config, MAX_DAYS_HENCE=30. But it is not reflecting.

0 Karma

woodcock
Esteemed Legend

Did you deploy to your INDEXERS and restart all splunk instances there? This will only apply to events you index AFTER the change (older broken events will "stay missing").

0 Karma

dsiob
Communicator

'Did you deploy to your INDEXERS' can you pls tell me how to do that.

0 Karma

dsiob
Communicator

i am uploading csv and using it without specify any index

0 Karma

woodcock
Esteemed Legend

You should change that back, create a C:\Program Files\Splunk
etc\system\local\props.conf
file and put in it this:

[default]
MAX_DAYS_HENCE=300

Then, assuming that you are using a splunk all-in-one instance on your personal computer, go to Settings -> Server controls -> Restart.

0 Karma

dsiob
Communicator

thanks Woodcock..its working!

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...