topic Re: Why am I getting error "Could not use regex to parse timestamp..." for timestamps before November 2010? in Splunk Search

Why am I getting error "Could not use regex to parse timestamp..." for timestamps before November 2010?

TangentTexan — Sat, 21 May 2016 16:31:11 GMT

Using Splunk 6.4.0 on Ubuntu Server

Trying to index a file that goes back in years. Working with the Timestamp to get it indexed correctly - I ran into a problem with it for Time Stamps before 11-30-2010, using this format: %Y-%m-%d %H:%M:%S %Z

Am I missing something? Thank you.

Sample Data: timedate, number

Could not use regex to parse timestamp from "2010-11-24 22:00:00 EST".
2010-11-24 22:00:00 EST,3.1
Could not use regex to parse timestamp from "2010-11-25 22:00:00 EST".
2010-11-25 22:00:00 EST,2.22
Could not use regex to parse timestamp from "2010-11-26 22:00:00 EST".
2010-11-26 22:00:00 EST,3.33
Could not use regex to parse timestamp from "2010-11-27 22:00:00 EST".
2010-11-27 22:00:00 EST,4.44
Could not use regex to parse timestamp from "2010-11-28 22:00:00 EST".
2010-11-28 22:00:00 EST,5.2
Could not use regex to parse timestamp from "2010-11-29 22:00:00 EST".
2010-11-29 22:00:00 EST,6.1
2010-11-30 22:00:00 EST,7.2
2010-12-01 22:00:00 EST,8.5
2010-12-02 22:00:00 EST,9.8
2010-12-03 22:00:00 EST,9.2
2010-12-04 22:00:00 EST,9.2
2010-12-05 22:00:00 EST,9.9

Re: Why am I getting error "Could not use regex to parse timestamp..." for timestamps before November 2010?

woodcock — Sat, 21 May 2016 19:06:07 GMT

Many times this is because the spacing between the date elements changed at some point. This is very hard to see and also a big pain because it requires you to switch from using the "easy" configurations to using datetime.xml (so that you can specify more than one time format. Very carefully check the spacing and you will probably find variation.

Re: Why am I getting error "Could not use regex to parse timestamp..." for timestamps before November 2010?

TangentTexan — Sat, 21 May 2016 22:35:30 GMT

I went through the standard process of verifying the data for any special characters or spaces... it was a no go.
I simplified the CSV and found all dates before 11-30-2010 would not get assigned the time stamp.
I think this is a bug. So I ended up excluding all log data before 1-1-2011 and stopped battling the timestamp. 🙂
I appreciate your time - thanks!

timedate,num
11/24/1996 22:00:00,1
11/25/1999 22:00:00,2
11/26/2000 22:00:00,3
11/27/2010 22:00:00,4
11/28/2010 22:00:00,5
11/29/2010 22:00:00,6
11/30/2010 22:00:00,7
12/01/2010 22:00:00,8
12/02/2010 22:00:00,9
12/03/2010 22:00:00,0
12/04/2010 22:00:00,10
12/05/2010 22:00:00,11

Re: Why am I getting error "Could not use regex to parse timestamp..." for timestamps before November 2010?

ddrillic — Sat, 21 May 2016 23:24:30 GMT

Looks fine to me -

base search 
| eval CREATION_DATE="2010-11-24 22:00:00 EST"
| eval xxxx=strptime(CREATION_DATE,"%Y-%m-%d %H:%M:%S %Z")

returns xxxx as 1290654000.000000 which is Thu, 25 Nov 2010 03:00:00 GMT

Re: Why am I getting error "Could not use regex to parse timestamp..." for timestamps before November 2010?

TangentTexan — Sun, 22 May 2016 00:30:45 GMT

I was trying to process the time at index time as I am trying to do a fresh index data. as new data gets entered this is not an issue. Just the older data.

Thanks

Re: Why am I getting error "Could not use regex to parse timestamp..." for timestamps before November 2010?

antlefebvre — Tue, 29 Sep 2020 22:33:55 GMT

I know this is a dated question, but ran into the same issue. The solution for me was to increase MAX_DAYS_AGO in props.conf. Looks like based on your description of it not working for 6 year old data this was your issue as well. Since Splunk by default ignores dates over 5 1/2ish years old.

From the docs site:

Specifies the maximum number of days in the past, from the current date, that an extracted date can be valid.
For example, if MAX_DAYS_AGO = 10, Splunk software ignores dates older than 10 days from the current date and instead either uses the timestamp of the previous event, or uses the current index time of the event if it cannot determine a timestamp in the previous event.

The maximum settable number of days in the past is 10951.

Defaults to 2000 days
Note: If you have data that is more than 2000 days old, increase this setting.