Getting Data In

Timestamp sometimes not recognized in batch

Bradysmall
Explorer

I have files that are simple CSV.. using a batch to get them into splunk.

Logs look like this..

12-06-2012 23:58:53.738,JBSP_P03,2_SPORTS/WIDEVID,CAM_05,Take,J_LIGHT2,OK

in my props, I have

MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 999999
KV_MODE = None
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%N

50% of the events get timestamped correctly.

The other 50% get stamped for when the file got copied into the batch folder..

12-06-2012 23:58:53.738,JBSP_P03,2_SPORTS/WIDEVID,CAM_05,Take,J_LIGHT2,OK - Stamped in splunk correctly.

10-08-2004 23:07:27.156,TGQC32N,3_NON/AES1/2,PRFD1912,Take Path,8191,OK - Stamped incorrectly in splunk.

0 Karma
1 Solution

emiller42
Motivator

You may need to add a MAX_DAYS_AGO line to your props.conf stanza. By default, Splunk will throw out any extracted timestamps more than 2000 days old. Your example is past that threshold, which is why it used index time.

from the props.conf spec

MAX_DAYS_AGO = integer
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

View solution in original post

Bradysmall
Explorer

Thanks. I stumbled into something similar and was just in the middle of changing that variable to check it on myself and it worked.

0 Karma

emiller42
Motivator

You may need to add a MAX_DAYS_AGO line to your props.conf stanza. By default, Splunk will throw out any extracted timestamps more than 2000 days old. Your example is past that threshold, which is why it used index time.

from the props.conf spec

MAX_DAYS_AGO = integer
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

Bradysmall
Explorer

Thanks. I stumbled into something similar and was just in the middle of changing that variable to check it on myself and it worked.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...