topic Re: Does Splunk log falling back to automatic timestamp extraction? in Getting Data In

Does Splunk log falling back to automatic timestamp extraction?

Graham_Hanningt — Thu, 31 Oct 2019 02:23:26 GMT

After Splunk (I'm using 7.3.0) has indexed an event, is there any way to tell whether:

Splunk successfully used the TIME_FORMAT et al configuration settings in props.conf to extract the timestamp, or
Splunk tried but failed to extract the timestamp using the configuration settings, and fell back to using its built-in automatic timestamp extraction

In either case, the timestamp in the indexed event is correct, so there's no way to tell from the indexed event itself.

I have read, among other Splunk documentation, the text following the heading "How Splunk software assigns timestamps".

I note that this documentation does not specifically cover the situation that I am describing: the documentation does not explain that Splunk falls back on automatic timestamp extraction after attempting to extract a timestamp using TIME_FORMAT. Rather, the documentation states:

If no TIME_FORMAT was configured for the data, Splunk software attempts to automatically identify a time or date in the event itself

"If no TIME_FORMAT was configured": in an example of the situation that I am describing, I had configured TIME_FORMAT, but I had failed to configure the related TIME_PREFIX. Perhaps I'm just nitpicking, splitting hairs. Moving on...

From the Splunk docs topic "Configure timestamp recognition", with my additional highlighting:

If you don't set TIME_PREFIX but you do set TIME_FORMAT, the timestamp must appear at the very start of each event; otherwise, Splunk software will not be able to process the formatting instructions, and every event will contain a warning about the inability to use strptime. (It's possible that you will still end up with a valid timestamp, based on how Splunk software attempts to recover from the problem.)

In practice, to see those strptime warnings, I use the "upload" option in Splunk Web: the "preview" shows those warnings.

However, after indexing, I can't see those warnings. The following search finds no results:

index=_internal strptime

Then again, rereading that documentation topic:

every event will contain a warning about the inability to use strptime

The event "will contain a warning"? Contain? As in, the indexed event will contain a field with that warning? I don't see any such warning in the indexed events.

What am I missing? How do I see those warnings after indexing? For example, do I need to configure the logging level of a particular log channel to get Splunk to log such warnings? Or is there a field with this warning in the indexed events that I'm somehow overlooking?

Re: Does Splunk log falling back to automatic timestamp extraction?

richgalloway — Thu, 31 Oct 2019 02:37:59 GMT

I suggest you submit feedback on that documentation page as it's misleading. If strptime fails a "Failed to parse timestamp" warning message will be written to splunkd.log, but will NOT be contained in the event itself. Unfortunately, there is no way to tie the log message to the specific event that triggered it, just the sourcetype.

Re: Does Splunk log falling back to automatic timestamp extraction?

Graham_Hanningt — Thu, 31 Oct 2019 03:05:31 GMT

Thanks for your answer!

Re:

I suggest you submit feedback on that documentation page as it's misleading.

Will do.

Re:

If strptime fails

Thank you! Yes, the following search:

index=_internal "Failed to parse timestamp"

finds a few instances of that warning. I took the documentation too literally when it referred to a "warning about the inability to use strptime". I was looking for a warning that contained the string "strptime".

However, while interesting, those instances don't cover the specific situation I describe in my question: where Splunk successfully extracts the timestamp, but only after failing to extract the timestamp using TIME_FORMAT. Does that situation get logged?

Re: Does Splunk log falling back to automatic timestamp extraction?

Graham_Hanningt — Thu, 31 Oct 2019 03:32:41 GMT

For example data and configuration settings matching this situation, see my recent question "Configured vs automatic extraction for timestamps in ISO 8601 extended format?", but imagine that I hadn't specified TIME_PREFIX.

In fact, that is exactly what I am embarrassed to confess I was doing: I was specifying TIME_FORMAT without TIME_PREFIX. And I thought that this was working, because Splunk extracted the timestamps correctly. However, the "Could not use strptime to parse timestamp" warnings that I saw when previewing data before uploading it via Splunk Web made me revisit the documentation, and realize I was wrong. What was actually happening: my TIME_FORMAT configuration (at that point, sans TIME_PREFIX) was failing, and Splunk was falling back on its automatic timestamp extraction, which was succeeding. I want to know about this situation, without using the "upload" option in Splunk Web! I'm grateful for the successful fallback, but I want to know about it.

Re: Does Splunk log falling back to automatic timestamp extraction?

Graham_Hanningt — Mon, 04 Nov 2019 06:14:15 GMT

Any more thoughts on this? I'd like to use the Splunk docs comment that you recommended I submit as an excuse to suggest an enhancement (I don't have a Splunk support contract) to log this situation: that is, where time stamp extraction succeeds, but not because of TIME_FORMAT.

I'm holding off submitting that comment in case you come back with a comment showing me that this situation is logged, after all.

If I don't hear back from you, then, based on your answer and subsequent comments, I believe that the answer to my original question ("Does Splunk log falling back to automatic timestamp extraction?") is "no". I'll add a comment to your answer to that effect, and, with thanks for your input, accept your answer.

Re: Does Splunk log falling back to automatic timestamp extraction?

richgalloway — Thu, 07 Nov 2019 13:36:28 GMT

I'm not aware of an explicit log that says fallback parsing was used. It's just part of the normal processing of timestamps.
Be sure to submit feedback on the docs page, rather than a comment. Feedback is sent to the Docs team and is more likely to receive a response.

Re: Does Splunk log falling back to automatic timestamp extraction?

Graham_Hanningt — Fri, 08 Nov 2019 05:03:10 GMT

Thanks very much for your time on this question, much appreciated. Here's a copy of the feedback I submitted on that docs page

Re: the following text in this Splunk docs topic ("Configure timestamp recognition"):

every event will contain a warning about the inability to use strptime

I quoted that text in a question on Splunk Answers, "Does Splunk log falling back to automatic timestamp extraction?" (https://answers.splunk.com/answers/779316).

Prominent Splunk user rich_galloway [apologies for this typo: I incorrectly inserted an underscore in your user name] answered, "I suggest you submit feedback on that documentation page as it's misleading."

Hence this feedback.

Could you please either correct that text, or clarify how "every event will contain a warning"? Specifically, I question the use of the word "contain".

For details, please see the related comments on that question and answer in Splunk Answers. I'm about to add a comment to rich's answer requesting a related enhancement.

Regards,
Graham Hannington

Re: Does Splunk log falling back to automatic timestamp extraction?

Graham_Hanningt — Fri, 08 Nov 2019 05:05:06 GMT

Based on this answer and subsequent comments, the answer to the question "Does Splunk log falling back to automatic timestamp extraction?" is "No".

Re: Does Splunk log falling back to automatic timestamp extraction?

Graham_Hanningt — Fri, 08 Nov 2019 05:11:57 GMT

This answer means that Splunk can silently fail. If Splunk fails to extract an event timestamp using TIME_FORMAT, then it can successfully extract a timestamp by falling back to automatic timestamp extraction. The problem: the timestamp that Splunk automatic extraction identifies might not be the timestamp that the Splunk developer intended to be used as the event timestamp. This situation is not logged.

I would like to see this situation optionally logged. Perhaps, a LOG_TIME_FORMAT_FAIL = <boolean> setting (default: false) in props.conf?