Does the HTTP Event Collector API support events with arbitrary metadata?

[yeungdarea]

According to the "Format events for HTTP Event Collector" document, I can send time, host, source, sourcetype and index.

I would like to send additional event metadata. Is this possible?

Given I'm running Splunk 6.4.2 with an HTTP Event Collector,
When I send an event with a metadata key called foo with the value bar:

curl -k -vv -H "Content-Type: application/json" -H "Authorization: Splunk XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" http://splunk:8088/services/collector/event -d '{"event": "hello world", "foo": "bar"}'

Then, I get this response:

< HTTP/1.1 400 Bad Request
< Date: Tue, 09 Aug 2016 05:26:47 GMT
< Content-Type: application/json; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 27
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
< Server: Splunkd
< 
* Connection #0 to host 172.25.0.3 left intact
{"text":"No data","code":5}%

I was hoping for a 200 OK and to see my event with the "foo" metadata.

[gblock_splunk · Accepted solution]

@yeungdarea today HEC will not allow you to pass arbitrary metadata fields. There is something coming soon in HEC which will allow this and should ultimately make it to the Docker driver. For the Docker driver, the only option today is to add labels which will show up in the JSON as you observed, or you can explore extracting fields other ways.

[yeungdarea]

We are looking forward to trying out this new feature. Would love to hear more details.

[gblock_splunk]

Great! Email me and I can tell you more: gblock@splunk.com

[Jeremiah]

You can't send additional metadata, but you can certainly include your metadata as json within the event. You also should look at using the new raw endpoint:

http://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fraw

"Send raw data directly to the HTTP Event Collector. This endpoint allows one or more raw events to be sent in a single request. All events are parsed using the standard Splunk software pipeline, which includes breaking rules and timestamp extraction."

Since you can now apply props to the data you should be able to create extractions that add indexed fields.

[yeungdarea]

Thank you. Is there any consideration of this feature in Splunk's roadmap?

We are trying to build something generic, that works with docker and helps us get logs to Splunk. We want this to be something that "forwards and tags" logs, rather than something that "wraps logs in an envelope with tags" or "parses then merges with tags".

Forwarding is attractive because it means developers that use our log forwarder can expect that if they write something to STDOUT, it will go to Splunk that way. This means developers can be in control of which sourcetype they use. It means our component is of lower complexity, and we don't have to explain how we rewrite log events.

[yeungdarea]

We are evaluating inserting KV pairs in the source field, and providing Splunk with a configuration snippet that allows us to extract these fields at search time.

It would be much nicer if there was a way to do this in the HEC API, so we didn't need to configure anything.

[Jeremiah]

Take a look at these two links, if you haven't seem them already. They cover the Splunk docker driver, which uses the HEC.

https://docs.docker.com/engine/admin/logging/splunk/
http://blogs.splunk.com/2015/12/16/splunk-logging-driver-for-docker/

You'll see it works nicely with Docker, but does wrap the events as you say.

[mdub_rea]

I would also like to be able to attach meta-data to log events sent via the HEC.

My use case is logs from Docker containers. I want to pass through log-lines from each container, intact, and optionally specify a "source type" to tell Splunk how to parse them (which rules out transforming the lines on their way to Splunk). But, I also want to capture metadata such as container-name, e.g.

{
  "time": ...,
  "source": ...,
  "event": "192.168.0.1 fnord:/api/blah - 42.3 admin yup garbage",
  "sourcetype": "my-custom-reverse-proxy-log-format",
  "meta": {
    "stack": "myapp-demo",
    "container": {
      "name": "revproxy",
      "id": "4b6771ca97e3"
    }
  }
}

Is this possible?