Getting Data In

Timestamp sometimes not recognized in batch

Bradysmall
Explorer

I have files that are simple CSV.. using a batch to get them into splunk.

Logs look like this..

12-06-2012 23:58:53.738,JBSP_P03,2_SPORTS/WIDEVID,CAM_05,Take,J_LIGHT2,OK

in my props, I have

MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 999999
KV_MODE = None
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%N

50% of the events get timestamped correctly.

The other 50% get stamped for when the file got copied into the batch folder..

12-06-2012 23:58:53.738,JBSP_P03,2_SPORTS/WIDEVID,CAM_05,Take,J_LIGHT2,OK - Stamped in splunk correctly.

10-08-2004 23:07:27.156,TGQC32N,3_NON/AES1/2,PRFD1912,Take Path,8191,OK - Stamped incorrectly in splunk.

0 Karma
1 Solution

emiller42
Motivator

You may need to add a MAX_DAYS_AGO line to your props.conf stanza. By default, Splunk will throw out any extracted timestamps more than 2000 days old. Your example is past that threshold, which is why it used index time.

from the props.conf spec

MAX_DAYS_AGO = integer
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

View solution in original post

Bradysmall
Explorer

Thanks. I stumbled into something similar and was just in the middle of changing that variable to check it on myself and it worked.

0 Karma

emiller42
Motivator

You may need to add a MAX_DAYS_AGO line to your props.conf stanza. By default, Splunk will throw out any extracted timestamps more than 2000 days old. Your example is past that threshold, which is why it used index time.

from the props.conf spec

MAX_DAYS_AGO = integer
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

Bradysmall
Explorer

Thanks. I stumbled into something similar and was just in the middle of changing that variable to check it on myself and it worked.

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...

Customer success is front and center at .conf25

Hi Splunkers, If you are not able to be at .conf25 in person, you can still learn about all the latest news ...