Splunk Search

What would cause the "date_*" fields to be missing from an event?

Super Champion

Can anyone tell me the reasons why timestartpos, timeendpos, and all the date_* fields would be missing from an event?

Here is a sample event with this issue:

3931|20101012125332|APP.STARTUP|system|3|INFO|Thread[PropertiesExtensionLoaderService,5,jboss] APPService / stopIt(): Stopping PropertiesExtensionLoaderService service...

This event should be assigned the date: 2010-10-12 12:53:32
But the event shows up in Splunk with the date: 10/8/10 1:56:22.000 PM.

There are several events that were assigned this incorrect timestamp. The souretype this event is assigned to has an explicit TIME_PREFIX and TIME_FORMAT that works fine. And the config hasn't changed. (I just recently did a "clean all" and reindexed everything, so it seems odd to me that some of the events are incorrect and others are fine; when there have been no config changes since the reindex.)

I looked in the original log file, and I can find "20101008135622" as a timestamp many events before the even shown in the example; But this time stamp doesn't immediately precede the example event; and the events in between this timestamp and my actual event all seem to have the correct timestamps themselves. Very odd.

Any ideas? If the time parser goes out to lunch, does that also keep the "date_*" fields from being created?


Server/Version info:

I found this issue on a new Splunk forwarder I'm in the process of setting up. I'm still testing it now, so I haven't enabled the forwarding yet; it's still running under the demo license in a local-only mode. The system is Win2k8R2 running Splunk 4.1.5.

Update:

I don't see any timestamping issues related to this in the _internal index, but I do see a number of messages like this:

10-12-2010 12:53:32.619 WARN DateParserVerbose - The TIME_FORMAT specified is matching timestamps (Tue Oct 12 12:53:32 2010) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.

I've asked a new question about this: timestamp outside of the acceptable time window — Huh?

Splunk Employee
Splunk Employee

I just tried this event and it should have worked find out of the box, without custom timestamping...

$splunk cmd parsetest "3931|20101012125332|APP.STARTUP|system|3|INFO|Thread..."
...     
Parsed: Tue Oct 12 12:53:32 2010
UTC Time:       1286913212

I don't know what your TIME_FORMAT values are, but if similar events work fine, then that's probably not the issue.

There are a few other possibly causes. Timestamps still have to pass the MAX checks, two of which could be relevant here, and default to:

MAX_DAYS_HENCE=2
MAX_DIFF_SECS_AGO=3600

if the event in question was more than 2 days in the future (somewhat unlikely) or if it was more than an hour before the previous event (possible with syslog being out of order). If your timestamps are wildly out of order, consider increasing this MAX_DIFF_SECS_AGO.

0 Karma

Super Champion

See my comment about MAX_* settings on this question: http://answers.splunk.com/questions/7924/matching-timestamp-outside-of-the-acceptable-time-window. I understand how out-of-order timestamp can cause issues, but I don't think that's what's happening here.

0 Karma

Splunk Employee
Splunk Employee

Those fields are only present when a timestamp was successfully parsed from your event. They represent the state of the parser and the literal values in the event, not an interpretation of the time in the time zone of any Splunk instance.

Super Champion

So would it be fairly save to assume that if these fields are missing from and event, then the timestamp recognition processed failed (for lack of a better term) for that event? And if so, are there any common reasons why this would happen?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!