Splunk Search

Splunk does not pickup custom timestamp prior to year 2012

raja21
Explorer

Splunk Version: 7.1

I have a custom time stamp field in my JSON records in this format, "_timestamp"="1/3/2013 10:12:56.000 PM".
On uploading the record Splunk picks up the custom timestamp and assigns it to _time, however when the year is before 2012 i.e.
"_timestamp"="1/3/2012 10:12:56.000 PM" or "_timestamp"="1/3/2011 10:12:56.000 PM" splunk throws an error that it cannot use regex to parse the timestamp.
I have been beating my head around this and would love to know the solution for this.

P.S. i have not changed any configuration in props.conf, splunk automatically picks up the custom timestamp as it is in the exact format.

Tags (1)
0 Karma
1 Solution

FrankVl
Ultra Champion

I guess that behavior is because by default MAX_DAYS_AGO is set to 2000, which is ~5,5 years, which indeed at this moment results in Splunk accepting timestamps from 2013, but not from 2012.

So if you want to indeed ingest that old data, you will have to configure Splunk to allow that.

View solution in original post

FrankVl
Ultra Champion

I guess that behavior is because by default MAX_DAYS_AGO is set to 2000, which is ~5,5 years, which indeed at this moment results in Splunk accepting timestamps from 2013, but not from 2012.

So if you want to indeed ingest that old data, you will have to configure Splunk to allow that.

raja21
Explorer

Hi @FrankVI thanks for your suggestion.
Can you please tell me how and where should i set MAX_DAYS_AGO?

0 Karma

niketn
Legend

It should be setup in props.conf for the sourcetype you are trying to add.

If you are testing with sample data (Upload a test file), you can test this out while setting the sourcetype under the Advanced Settings section by manually typing the MAX_DAYS_AGO and the number of days to identify the oldest event.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

raja21
Explorer

@niketnilay I got it working, thanks 🙂

0 Karma

niketn
Legend

@raja21, glad you found it working. Do up vote the comments that helped 🙂

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

niketn
Legend

@raja21 look into MAX_DAYS_AGO as suggested by FrankVI. Even while uploading couple of dummy events you can check the behavior whether correct timestamp is identified by changing MAX_DAYS_AGO or not.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

raja21
Explorer

Can you please tell me how and where should i set MAX_DAYS_AGO?

0 Karma

xpac
SplunkTrust
SplunkTrust

Best practice hint: Always set up timestamp recognition for your data, don't let Splunk guess. It will not only save you trouble like this, but maybe even more in the future - because Splunk might recognize that timestamp as "day 1 of month 3", as well as "day 3 of month 1".

0 Karma

raja21
Explorer

I'm practicing the standard format of %m/%d/%y H:M:S.000 AM/PM so it should always pick it up in the correct format i guess.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...