topic Re: Why does Splunk recognize the timestamp only for specific dates? in Splunk Search

Why does Splunk recognize the timestamp only for specific dates?

atemourt — Fri, 16 Mar 2018 10:29:11 GMT

Hello,

I have a csv file with data from 2010 until 2017.

Splunk seems to parse the timestamp correctly for most of the data but when the date is from 2010 or 2011 or 2012, I see the message: Could not use timestamp to parse the data from "".
i.e. Could not use timestamp to parse the data from "1/21/2010".

The format of date in the csv file is month/day/year.

Why does Splunk recognize the timestamp when the date is 1/20/2017 23:00:00 PM but it doesn't recognizes the timestamp when the date is 1/21/2010 11:00:00 AM?

Sample of data:
Date Type Latitude Longitude Id
1/21/2010 11:00 Dry Cargo 39.3869634 22.9385489 29
1/22/2010 8:00 Dry Cargo 39.3675609 22.9491659 30
1/23/2010 13:30 Dry Cargo 39.367539 22.9229295 31
1/24/2010 9:00 Refrigerated Cargo 39.3686508 22.9414365 32
1/26/2010 18:00 Dry Cargo 39.3766097 22.9603403 33
1/26/2010 17:00 Dry Cargo 39.3557886 22.9581058 34
1/27/2010 10:00 Refrigerated Cargo 39.3799523 22.9232278 35
1/27/2010 12:00 Dry Cargo 39.3647131 22.9517557 36

Thank you in advance!

Re: Why does Splunk recognize the timestamp only for specific dates?

tiagofbmm — Fri, 16 Mar 2018 13:02:54 GMT

Are you ingesting that file somehow or just inputing it as a lookup?

Re: Why does Splunk recognize the timestamp only for specific dates?

skoelpin — Fri, 16 Mar 2018 13:05:15 GMT

You need to set base configs which tell Splunk how to read the timestamp

Add this to your props.conf and restart the splunkd service

[YOUR_SOURCETYPE]
TIME_PREFIX=^
TIME_FORMAT=%m/%e/%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD=18

Re: Why does Splunk recognize the timestamp only for specific dates?

atemourt — Fri, 16 Mar 2018 13:08:49 GMT

I uploaded the csv file from my computer.

Re: Why does Splunk recognize the timestamp only for specific dates?

richgalloway — Tue, 29 Sep 2020 18:31:09 GMT

What is your MAX_DAYS_AGO setting? I would expect a different error message if this was the cause, but it's worth changing it to 5000 or so to see if it helps. The default setting is 2000, which means Splunk will reject timestamps more than 5 years old.

Re: Why does Splunk recognize the timestamp only for specific dates?

atemourt — Tue, 29 Sep 2020 18:34:03 GMT

Thanks for the advise.
I have set the MAX_DAYS_AGO to 5000 in props.conf.
Actually, my props.conf is:
[data]
DATETIME_CONFIG =
MAX_DAYS_AGO = 5000
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = Date
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

However, it doesn't work. Splunk still doesn't recognize dates from 2010, 2011 and 2012. 😞

Re: Why does Splunk recognize the timestamp only for specific dates?

skoelpin — Sun, 18 Mar 2018 23:10:19 GMT

It's because you don't have TIME_PREFIX or TIME_FORMAT set.. I gave you the correct stanza in my answer above..

Re: Why does Splunk recognize the timestamp only for specific dates?

atemourt — Mon, 19 Mar 2018 09:04:12 GMT

Hello skoelpin,

Thank you for the answer.

I have tried what you suggested, but Splunk cannot read the timestamp.
I still see the message: Could not use timestamp to parse the data from "".

Is there anything else that I can try?

Re: Why does Splunk recognize the timestamp only for specific dates?

richgalloway — Tue, 20 Mar 2018 02:04:59 GMT

Did you restart Splunk after modifying the config file?

Re: Why does Splunk recognize the timestamp only for specific dates?

atemourt — Tue, 20 Mar 2018 09:19:32 GMT

Yes, I restarted Splunk.
Every time I do a change in the conf files, I restart Splunk.