All Apps and Add-ons

dbconnect2 timestamp parsing

robertlabrie
Path Finder

DBX2. Latest versions of both (as of today). Wn7 x64. Data is in MySQL.

timestamp="2007-10-02 20:31:47"

That's an actual timestamp in my data. Field is a varchar.

timestamp format in the input config is: yyyy-MM-dd HH:mm:ss

The value splunk got for _time 2015-09-19T20:31:47.000-04:00

OUtput timestamp format is: YYYY-MM-dd HH:mm:ss
I also tried that format in the input config.'

The only thing that gets parsed reliably is the minutes and seconds. Everything else is wrong. DBX2 is 1000x better than DBx1. Easy to use, looks great. Why is time parsing so hard? Is it not SimpleDateFormat? Is it python? Do I need to use the percents?

0 Karma
1 Solution

woodcock
Esteemed Legend

Check for a log like the following in splunkd.log:

mm-dd-yyyy HH:MM:SS.### +0000 WARN  DateParserVerbose - A possible timestamp match (--------------------) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.  Context="source::--------------------"

From the props.conf docs:

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

So, set MAX_DAYS_AGO to something high enough in props.conf and you should be good to go.

View solution in original post

0 Karma

woodcock
Esteemed Legend

Check for a log like the following in splunkd.log:

mm-dd-yyyy HH:MM:SS.### +0000 WARN  DateParserVerbose - A possible timestamp match (--------------------) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.  Context="source::--------------------"

From the props.conf docs:

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

So, set MAX_DAYS_AGO to something high enough in props.conf and you should be good to go.

0 Karma

robertlabrie
Path Finder

I didn't have that event in the logs, but when I limited my results to 2015 the time parsed correctly. Set MAX_DAYS_AGO to 10000 and now it works fine. Thanks for your help. Wish splunk would just not index the data instead of consuming it wrong....

0 Karma

woodcock
Esteemed Legend

I should not have mentioned the log because that would only have been generated if Splunk had read the timedate correctly and then ignored the log. In your case (I am not sure what causes the difference), Splunk instead assumes that the date (being so old?) is not in the proper format/place so it just uses some other time (previous event or current time) to date the event. I see things happen both ways with very old dates and it would be nice if I could figure out why sometimes it handles it one way vs. the other.

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...