JSON timestamps not parsed via HTTP Event Collector

[bradserbuddy]

Here are some of the values I am using for my JSON source type:

MAX_TIMESTAMP_LOOKAHEAD = 1000 (as we have long JSON input)
TIME_FORMAT = %FT%T.%3Q
TIME_PREFIX = Timestamp\"\s:\s\"

I've successfully imported the JSON from a file with the above source type values, but, for some reason, when coming in through my HTTP Event Collector, the timestamp isn't picked up (that is, _time is not set to the timestamp).

I've restarted the server, tried different values for TIME_PREFIX (for instance not encoding the quotes, and dropping the \s regex) and TIME_FORMAT (for instance %Y-%m-%dT%H:%M:%S), and removed the KV_MODE=json to no avail.

Am I misunderstanding the relationship between timestamp parsing and _time? Is there something else I need to do to get my source type to work with my HTTP Event Collector? Are there additional troubleshooting steps/tools I can use to help track down what's going on?

Thanks,
Brad

[gblock_splunk · Accepted solution]

Our JSON endpoint for HTTP Event Collector does not do timestamp extraction. Our JSON endpoint expects events sent using a lightweight structure, our JSON Event Protocol. The plus side is your "event" payload can be any JSON (or not) / can have whitespace, newlines, etc and it will still come as a single event. For example you could also have a Java stack trace and it will easily go in as a single event. For the timestamp, you have to specify "time" explicitly outside the payload in the event envelope in epoch format i.e.

{
"time": 1426279439,
"host": "localhost",
"source": "datasource",
"sourcetype": "txt",
"index": "main",
"event": { "hello": "world" }
}

In Splunk Cloud / Splunk 6.4 you have a different option, our new raw endpoint. With raw you can send an arbitrary payload to HEC (/services/collector/raw) and we will honor breaking rules and do timestamp extraction. It should I believe meet your need.

[bradserbuddy]

I switched to using the Splunk-provided generic_single_line source type, and that doesn't work either. I am testing by POSTing to the collector, which has no TIME_PREFIX, and only posting the timestamp itself. It appears that I can only set the time via the "time" property on the enclosing JSON itself as detailed here: http://dev.splunk.com/view/SP-CAAAE6P. Is that true?