Getting Data In

Can we process the timestamp in an event sent to the HTTP event collector?

Jeremiah
Motivator

The HTTP event collector supports an optional timestamp:

{
    "time": "1426279439", 
    "host": "localhost",
    "source": "datasource",
    "sourcetype": "txt",
    "index": "main",
    "event": { "hello": "world" }
}

But what if I want to process the timestamp directly from the event, like this:

   {
        "host": "localhost",
        "source": "datasource",
        "sourcetype": "txt",
        "index": "main",
        "event": { "message": "9/29/2015 13:00:00 hello world" }
    }

Can I do this? It seems like Splunk skips timestamp extraction for events posted to the collector, regardless of sourcetype.

1 Solution

richgalloway
SplunkTrust
SplunkTrust

According to the presentation at .conf2015, the HTTP Event Collector will only look for event timestamps in the "time" field, which must be in epoch form.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

According to the presentation at .conf2015, the HTTP Event Collector will only look for event timestamps in the "time" field, which must be in epoch form.

---
If this reply helps you, Karma would be appreciated.

yannK
Splunk Employee
Splunk Employee
0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

Yes this is correct, use "time".

0 Karma

Graham_Hanningt
Builder

According to the Splunk Dev page "About the JSON event protocol in HTTP Event Collector":

The default time format is epoch time format, in the format <sec>.<ms>. For example, 1433188255.5 indicates 1433188255 seconds and 5 microseconds

"5 microseconds" is wrong. In this context, that .5 indicates half a second. And ms is the abbreviation for milliseconds, not microseconds. The abbreviation for microseconds is (stand back, I'm going to attempt a mu) μs. I would be happy to learn that the event time precision is microseconds, but I suspect (as per ms) that it's milliseconds (is it?).

As a trial user only, I could find no more direct method of feedback than reporting this via email to devinfo@splunk.com, but I've yet to get a (non-automated) reply, so I thought I'd mention it here. Please feel free to direct me to use some other feedback method for this type of comment.

On a related issue, I'm currently in denial about what it appears I have to do to get Splunk to display event times in ISO 8601 format.

Graham_Hanningt
Builder

In my previous comment, I used the "Hyperlink" toolbar button to convert that "About..." page title into a hyperlink. It didn't work.

Before submitting that comment, I entered the comment as an answer (with no intention of submitting it as answer) so that I could preview it, because I cannot see how to preview comments (although I was aware that comments might only support a subset of the markdown supported by answers). I couldn't get a hyperlink to work there, either: neither using the "reference"-style syntax generated by the Hyperlink toolbar, nor the more direct "link text in square brackets followed by URL in parentheses" syntax specified by the Splunk Answers Markdown Syntax web page.

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

Graham, you are correct, that is milliseconds. This would be 500 ms as everything after the decimal / after 10 digits is milliseconds. I'll get the docs updated. Thanks for reporting.

0 Karma

Graham_Hanningt
Builder

Thanks, @gblock_splunk.

From that same Splunk Dev page:

 "time": "1426279439"

Why is the time value enclosed in quotes? It's a number, not a string.

Those quotes are not required by JSON, and not necessary in practice; in testing, I omitted the quotes without even thinking about it, and it "worked":

{"text":"Success","code":0}

Note: no quotes around the 0 value of "code"  (trying for an emoji smile there).

gblock_splunk
Splunk Employee
Splunk Employee

It should not be quoted, that is a bug in the docs. Will be fixed.

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

No problem @Graham_Hannington thank you for taking the time to report this.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...