Using Splunk 6.4.0 on Ubuntu Server
Trying to index a file that goes back in years. Working with the Timestamp to get it indexed correctly - I ran into a problem with it for Time Stamps before 11-30-2010, using this format:
%Y-%m-%d %H:%M:%S %Z
Am I missing something? Thank you.
Sample Data: timedate, number
Could not use regex to parse timestamp from "2010-11-24 22:00:00 EST".
2010-11-24 22:00:00 EST,3.1
Could not use regex to parse timestamp from "2010-11-25 22:00:00 EST".
2010-11-25 22:00:00 EST,2.22
Could not use regex to parse timestamp from "2010-11-26 22:00:00 EST".
2010-11-26 22:00:00 EST,3.33
Could not use regex to parse timestamp from "2010-11-27 22:00:00 EST".
2010-11-27 22:00:00 EST,4.44
Could not use regex to parse timestamp from "2010-11-28 22:00:00 EST".
2010-11-28 22:00:00 EST,5.2
Could not use regex to parse timestamp from "2010-11-29 22:00:00 EST".
2010-11-29 22:00:00 EST,6.1
2010-11-30 22:00:00 EST,7.2
2010-12-01 22:00:00 EST,8.5
2010-12-02 22:00:00 EST,9.8
2010-12-03 22:00:00 EST,9.2
2010-12-04 22:00:00 EST,9.2
2010-12-05 22:00:00 EST,9.9
Many times this is because the spacing between the date elements changed at some point. This is very hard to see and also a big pain because it requires you to switch from using the "easy" configurations to using
datetime.xml (so that you can specify more than one time format. Very carefully check the spacing and you will probably find variation.
I went through the standard process of verifying the data for any special characters or spaces... it was a no go.
I simplified the CSV and found all dates before 11-30-2010 would not get assigned the time stamp.
I think this is a bug. So I ended up excluding all log data before 1-1-2011 and stopped battling the timestamp. 🙂
I appreciate your time - thanks!
Looks fine to me -
base search | eval CREATION_DATE="2010-11-24 22:00:00 EST" | eval xxxx=strptime(CREATION_DATE,"%Y-%m-%d %H:%M:%S %Z")
returns xxxx as
1290654000.000000 which is
Thu, 25 Nov 2010 03:00:00 GMT
I was trying to process the time at index time as I am trying to do a fresh index data. as new data gets entered this is not an issue. Just the older data.
I know this is a dated question, but ran into the same issue. The solution for me was to increase MAXDAYSAGO in props.conf. Looks like based on your description of it not working for 6 year old data this was your issue as well. Since Splunk by default ignores dates over 5 1/2ish years old.
From the docs site:
Specifies the maximum number of days in the past, from the current date, that an extracted date can be valid.
For example, if MAXDAYSAGO = 10, Splunk software ignores dates older than 10 days from the current date and instead either uses the timestamp of the previous event, or uses the current index time of the event if it cannot determine a timestamp in the previous event.
The maximum settable number of days in the past is 10951.
Defaults to 2000 days
Note: If you have data that is more than 2000 days old, increase this setting.