Getting Data In

Can we process the timestamp in an event sent to the HTTP event collector?

Motivator

The HTTP event collector supports an optional timestamp:

{
    "time": "1426279439", 
    "host": "localhost",
    "source": "datasource",
    "sourcetype": "txt",
    "index": "main",
    "event": { "hello": "world" }
}

But what if I want to process the timestamp directly from the event, like this:

   {
        "host": "localhost",
        "source": "datasource",
        "sourcetype": "txt",
        "index": "main",
        "event": { "message": "9/29/2015 13:00:00 hello world" }
    }

Can I do this? It seems like Splunk skips timestamp extraction for events posted to the collector, regardless of sourcetype.

1 Solution

SplunkTrust
SplunkTrust

According to the presentation at .conf2015, the HTTP Event Collector will only look for event timestamps in the "time" field, which must be in epoch form.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

SplunkTrust
SplunkTrust

According to the presentation at .conf2015, the HTTP Event Collector will only look for event timestamps in the "time" field, which must be in epoch form.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Splunk Employee
Splunk Employee
0 Karma

Splunk Employee
Splunk Employee

Yes this is correct, use "time".

0 Karma

According to the Splunk Dev page "About the JSON event protocol in HTTP Event Collector":

The default time format is epoch time format, in the format <sec>.<ms>. For example, 1433188255.5 indicates 1433188255 seconds and 5 microseconds

"5 microseconds" is wrong. In this context, that .5 indicates half a second. And ms is the abbreviation for milliseconds, not microseconds. The abbreviation for microseconds is (stand back, I'm going to attempt a mu) μs. I would be happy to learn that the event time precision is microseconds, but I suspect (as per ms) that it's milliseconds (is it?).

As a trial user only, I could find no more direct method of feedback than reporting this via email to devinfo@splunk.com, but I've yet to get a (non-automated) reply, so I thought I'd mention it here. Please feel free to direct me to use some other feedback method for this type of comment.

On a related issue, I'm currently in denial about what it appears I have to do to get Splunk to display event times in ISO 8601 format.

In my previous comment, I used the "Hyperlink" toolbar button to convert that "About..." page title into a hyperlink. It didn't work.

Before submitting that comment, I entered the comment as an answer (with no intention of submitting it as answer) so that I could preview it, because I cannot see how to preview comments (although I was aware that comments might only support a subset of the markdown supported by answers). I couldn't get a hyperlink to work there, either: neither using the "reference"-style syntax generated by the Hyperlink toolbar, nor the more direct "link text in square brackets followed by URL in parentheses" syntax specified by the Splunk Answers Markdown Syntax web page.

0 Karma

Splunk Employee
Splunk Employee

Graham, you are correct, that is milliseconds. This would be 500 ms as everything after the decimal / after 10 digits is milliseconds. I'll get the docs updated. Thanks for reporting.

0 Karma

Thanks, @gblock_splunk.

From that same Splunk Dev page:

 "time": "1426279439"

Why is the time value enclosed in quotes? It's a number, not a string.

Those quotes are not required by JSON, and not necessary in practice; in testing, I omitted the quotes without even thinking about it, and it "worked":

{"text":"Success","code":0}

Note: no quotes around the 0 value of "code"  (trying for an emoji smile there).

Splunk Employee
Splunk Employee

It should not be quoted, that is a bug in the docs. Will be fixed.

0 Karma

Splunk Employee
Splunk Employee

No problem @Graham_Hannington thank you for taking the time to report this.

0 Karma