Getting Data In

Does the HTTP Event Collector API support events with arbitrary metadata?

yeungdarea
Explorer

According to the "Format events for HTTP Event Collector" document, I can send time, host, source, sourcetype and index.

I would like to send additional event metadata. Is this possible?

Given I'm running Splunk 6.4.2 with an HTTP Event Collector,
When I send an event with a metadata key called foo with the value bar:

curl -k -vv -H "Content-Type: application/json" -H "Authorization: Splunk XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" http://splunk:8088/services/collector/event -d '{"event": "hello world", "foo": "bar"}'

Then, I get this response:

< HTTP/1.1 400 Bad Request
< Date: Tue, 09 Aug 2016 05:26:47 GMT
< Content-Type: application/json; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 27
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
< Server: Splunkd
< 
* Connection #0 to host 172.25.0.3 left intact
{"text":"No data","code":5}%

I was hoping for a 200 OK and to see my event with the "foo" metadata.

Tags (1)
1 Solution

gblock_splunk
Splunk Employee
Splunk Employee

@yeungdarea today HEC will not allow you to pass arbitrary metadata fields. There is something coming soon in HEC which will allow this and should ultimately make it to the Docker driver. For the Docker driver, the only option today is to add labels which will show up in the JSON as you observed, or you can explore extracting fields other ways.

View solution in original post

gblock_splunk
Splunk Employee
Splunk Employee

@yeungdarea today HEC will not allow you to pass arbitrary metadata fields. There is something coming soon in HEC which will allow this and should ultimately make it to the Docker driver. For the Docker driver, the only option today is to add labels which will show up in the JSON as you observed, or you can explore extracting fields other ways.

yeungdarea
Explorer

We are looking forward to trying out this new feature. Would love to hear more details.

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

Great! Email me and I can tell you more: gblock@splunk.com

0 Karma

Jeremiah
Motivator

You can't send additional metadata, but you can certainly include your metadata as json within the event. You also should look at using the new raw endpoint:

http://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fraw

"Send raw data directly to the HTTP Event Collector. This endpoint allows one or more raw events to be sent in a single request. All events are parsed using the standard Splunk software pipeline, which includes breaking rules and timestamp extraction."

Since you can now apply props to the data you should be able to create extractions that add indexed fields.

yeungdarea
Explorer

Thank you. Is there any consideration of this feature in Splunk's roadmap?

We are trying to build something generic, that works with docker and helps us get logs to Splunk. We want this to be something that "forwards and tags" logs, rather than something that "wraps logs in an envelope with tags" or "parses then merges with tags".

Forwarding is attractive because it means developers that use our log forwarder can expect that if they write something to STDOUT, it will go to Splunk that way. This means developers can be in control of which sourcetype they use. It means our component is of lower complexity, and we don't have to explain how we rewrite log events.

0 Karma

yeungdarea
Explorer

We are evaluating inserting KV pairs in the source field, and providing Splunk with a configuration snippet that allows us to extract these fields at search time.

It would be much nicer if there was a way to do this in the HEC API, so we didn't need to configure anything.

0 Karma

Jeremiah
Motivator

Take a look at these two links, if you haven't seem them already. They cover the Splunk docker driver, which uses the HEC.

https://docs.docker.com/engine/admin/logging/splunk/
http://blogs.splunk.com/2015/12/16/splunk-logging-driver-for-docker/

You'll see it works nicely with Docker, but does wrap the events as you say.

0 Karma

mdub_rea
Engager

I would also like to be able to attach meta-data to log events sent via the HEC.

My use case is logs from Docker containers. I want to pass through log-lines from each container, intact, and optionally specify a "source type" to tell Splunk how to parse them (which rules out transforming the lines on their way to Splunk). But, I also want to capture metadata such as container-name, e.g.

{
  "time": ...,
  "source": ...,
  "event": "192.168.0.1 fnord:/api/blah - 42.3 admin yup garbage",
  "sourcetype": "my-custom-reverse-proxy-log-format",
  "meta": {
    "stack": "myapp-demo",
    "container": {
      "name": "revproxy",
      "id": "4b6771ca97e3"
    }
  }
}

Is this possible?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...