topic Re: Can we process the timestamp in an event sent to the HTTP event collector? in Getting Data In

Can we process the timestamp in an event sent to the HTTP event collector?

Jeremiah — Tue, 29 Sep 2015 13:41:18 GMT

The HTTP event collector supports an optional timestamp:

{
    "time": "1426279439", 
    "host": "localhost",
    "source": "datasource",
    "sourcetype": "txt",
    "index": "main",
    "event": { "hello": "world" }
}

But what if I want to process the timestamp directly from the event, like this:

   {
        "host": "localhost",
        "source": "datasource",
        "sourcetype": "txt",
        "index": "main",
        "event": { "message": "9/29/2015 13:00:00 hello world" }
    }

Can I do this? It seems like Splunk skips timestamp extraction for events posted to the collector, regardless of sourcetype.

Re: Can we process the timestamp in an event sent to the HTTP event collector?

richgalloway — Tue, 29 Sep 2015 13:49:01 GMT

According to the presentation at .conf2015, the HTTP Event Collector will only look for event timestamps in the "time" field, which must be in epoch form.

Re: Can we process the timestamp in an event sent to the HTTP event collector?

gblock_splunk — Thu, 01 Oct 2015 04:47:59 GMT

Yes this is correct, use "time".

Re: Can we process the timestamp in an event sent to the HTTP event collector?

Graham_Hanningt — Wed, 18 Nov 2015 03:03:59 GMT

According to the Splunk Dev page "About the JSON event protocol in HTTP Event Collector":

The default time format is epoch time format, in the format <sec>.<ms>. For example, 1433188255.5 indicates 1433188255 seconds and 5 microseconds

"5 microseconds" is wrong. In this context, that .5 indicates half a second. And ms is the abbreviation for milliseconds, not microseconds. The abbreviation for microseconds is (stand back, I'm going to attempt a mu) μs. I would be happy to learn that the event time precision is microseconds, but I suspect (as per ms) that it's milliseconds (is it?).

As a trial user only, I could find no more direct method of feedback than reporting this via email to devinfo@splunk.com, but I've yet to get a (non-automated) reply, so I thought I'd mention it here. Please feel free to direct me to use some other feedback method for this type of comment.

On a related issue, I'm currently in denial about what it appears I have to do to get Splunk to display event times in ISO 8601 format.

Re: Can we process the timestamp in an event sent to the HTTP event collector?

Graham_Hanningt — Wed, 18 Nov 2015 03:29:36 GMT

In my previous comment, I used the "Hyperlink" toolbar button to convert that "About..." page title into a hyperlink. It didn't work.

Before submitting that comment, I entered the comment as an answer (with no intention of submitting it as answer) so that I could preview it, because I cannot see how to preview comments (although I was aware that comments might only support a subset of the markdown supported by answers). I couldn't get a hyperlink to work there, either: neither using the "reference"-style syntax generated by the Hyperlink toolbar, nor the more direct "link text in square brackets followed by URL in parentheses" syntax specified by the Splunk Answers Markdown Syntax web page.

Re: Can we process the timestamp in an event sent to the HTTP event collector?

gblock_splunk — Wed, 18 Nov 2015 05:37:04 GMT

Graham, you are correct, that is milliseconds. This would be 500 ms as everything after the decimal / after 10 digits is milliseconds. I'll get the docs updated. Thanks for reporting.

Re: Can we process the timestamp in an event sent to the HTTP event collector?

Graham_Hanningt — Wed, 18 Nov 2015 07:43:16 GMT

Thanks, @gblock_splunk.

From that same Splunk Dev page:

 "time": "1426279439"

Why is the time value enclosed in quotes? It's a number, not a string.

Those quotes are not required by JSON, and not necessary in practice; in testing, I omitted the quotes without even thinking about it, and it "worked":

{"text":"Success","code":0}

Note: no quotes around the 0 value of "code"  (trying for an emoji smile there).

Re: Can we process the timestamp in an event sent to the HTTP event collector?

gblock_splunk — Wed, 18 Nov 2015 07:46:07 GMT

It should not be quoted, that is a bug in the docs. Will be fixed.

Re: Can we process the timestamp in an event sent to the HTTP event collector?

gblock_splunk — Wed, 18 Nov 2015 07:48:48 GMT

No problem @Graham_Hannington thank you for taking the time to report this.

Re: Can we process the timestamp in an event sent to the HTTP event collector?

yannK — Mon, 07 Mar 2016 23:54:46 GMT

more details here http://dev.splunk.com/view/event-collector/SP-CAAAE6M