Getting Data In
Highlighted

JSON timestamps not parsed via HTTP Event Collector

Engager

Here are some of the values I am using for my JSON source type:

MAXTIMESTAMPLOOKAHEAD = 1000 (as we have long JSON input)
TIMEFORMAT = %FT%T.%3Q
TIME
PREFIX = Timestamp\"\s:\s\"

I've successfully imported the JSON from a file with the above source type values, but, for some reason, when coming in through my HTTP Event Collector, the timestamp isn't picked up (that is, _time is not set to the timestamp).

I've restarted the server, tried different values for TIMEPREFIX (for instance not encoding the quotes, and dropping the \s regex) and TIMEFORMAT (for instance %Y-%m-%dT%H:%M:%S), and removed the KV_MODE=json to no avail.

Am I misunderstanding the relationship between timestamp parsing and _time? Is there something else I need to do to get my source type to work with my HTTP Event Collector? Are there additional troubleshooting steps/tools I can use to help track down what's going on?

Thanks,
Brad

Highlighted

Re: JSON timestamps not parsed via HTTP Event Collector

Engager

I switched to using the Splunk-provided genericsingleline source type, and that doesn't work either. I am testing by POSTing to the collector, which has no TIME_PREFIX, and only posting the timestamp itself. It appears that I can only set the time via the "time" property on the enclosing JSON itself as detailed here: http://dev.splunk.com/view/SP-CAAAE6P. Is that true?

0 Karma
Highlighted

Re: JSON timestamps not parsed via HTTP Event Collector

Splunk Employee
Splunk Employee

Our JSON endpoint for HTTP Event Collector does not do timestamp extraction. Our JSON endpoint expects events sent using a lightweight structure, our JSON Event Protocol. The plus side is your "event" payload can be any JSON (or not) / can have whitespace, newlines, etc and it will still come as a single event. For example you could also have a Java stack trace and it will easily go in as a single event. For the timestamp, you have to specify "time" explicitly outside the payload in the event envelope in epoch format i.e.

{
"time": 1426279439,
"host": "localhost",
"source": "datasource",
"sourcetype": "txt",
"index": "main",
"event": { "hello": "world" }
}

In Splunk Cloud / Splunk 6.4 you have a different option, our new raw endpoint. With raw you can send an arbitrary payload to HEC (/services/collector/raw) and we will honor breaking rules and do timestamp extraction. It should I believe meet your need.

View solution in original post