Splunk Search

What would cause the "date_*" fields to be missing from an event?

Lowell
Super Champion

Can anyone tell me the reasons why timestartpos, timeendpos, and all the date_* fields would be missing from an event?

Here is a sample event with this issue:

3931|20101012125332|APP.STARTUP|system|3|INFO|Thread[PropertiesExtensionLoaderService,5,jboss] APPService / stopIt(): Stopping PropertiesExtensionLoaderService service...

This event should be assigned the date: 2010-10-12 12:53:32
But the event shows up in Splunk with the date: 10/8/10 1:56:22.000 PM.

There are several events that were assigned this incorrect timestamp. The souretype this event is assigned to has an explicit TIME_PREFIX and TIME_FORMAT that works fine. And the config hasn't changed. (I just recently did a "clean all" and reindexed everything, so it seems odd to me that some of the events are incorrect and others are fine; when there have been no config changes since the reindex.)

I looked in the original log file, and I can find "20101008135622" as a timestamp many events before the even shown in the example; But this time stamp doesn't immediately precede the example event; and the events in between this timestamp and my actual event all seem to have the correct timestamps themselves. Very odd.

Any ideas? If the time parser goes out to lunch, does that also keep the "date_*" fields from being created?


Server/Version info:

I found this issue on a new Splunk forwarder I'm in the process of setting up. I'm still testing it now, so I haven't enabled the forwarding yet; it's still running under the demo license in a local-only mode. The system is Win2k8R2 running Splunk 4.1.5.

Update:

I don't see any timestamping issues related to this in the _internal index, but I do see a number of messages like this:

10-12-2010 12:53:32.619 WARN DateParserVerbose - The TIME_FORMAT specified is matching timestamps (Tue Oct 12 12:53:32 2010) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.

I've asked a new question about this: timestamp outside of the acceptable time window — Huh?

carasso
Splunk Employee
Splunk Employee

I just tried this event and it should have worked find out of the box, without custom timestamping...

$splunk cmd parsetest "3931|20101012125332|APP.STARTUP|system|3|INFO|Thread..."
...     
Parsed: Tue Oct 12 12:53:32 2010
UTC Time:       1286913212

I don't know what your TIME_FORMAT values are, but if similar events work fine, then that's probably not the issue.

There are a few other possibly causes. Timestamps still have to pass the MAX checks, two of which could be relevant here, and default to:

MAX_DAYS_HENCE=2
MAX_DIFF_SECS_AGO=3600

if the event in question was more than 2 days in the future (somewhat unlikely) or if it was more than an hour before the previous event (possible with syslog being out of order). If your timestamps are wildly out of order, consider increasing this MAX_DIFF_SECS_AGO.

0 Karma

Lowell
Super Champion

See my comment about MAX_* settings on this question: http://answers.splunk.com/questions/7924/matching-timestamp-outside-of-the-acceptable-time-window. I understand how out-of-order timestamp can cause issues, but I don't think that's what's happening here.

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

Those fields are only present when a timestamp was successfully parsed from your event. They represent the state of the parser and the literal values in the event, not an interpretation of the time in the time zone of any Splunk instance.

Lowell
Super Champion

So would it be fairly save to assume that if these fields are missing from and event, then the timestamp recognition processed failed (for lack of a better term) for that event? And if so, are there any common reasons why this would happen?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...