Getting Data In

Sending input data over HTTP

Explorer

Does Splunk support receiving a continual stream of input via an HTTP POST?

The reason I ask is the web server logs I want to index in Splunk is not accessible from the Splunk Server. I already have a tool which can relay server log events over HTTP so I think this could work.

Tags (2)
0 Karma

Re:

Does Splunk support receiving a continual stream of input via an HTTP POST?

No.

Not a continual (endless) stream.

You can "batch" (send multiple) events in a single HTTP POST, but there is a maximum limit to the size of an HTTP request.

The limit is set by max_content_length in limits.conf. The default value, as of Splunk 6.4, is 1000000 bytes (~ 1 MB).

Exceeding that limit results in the HTTP response error code 413 (request entity too large).

The Splunk documentation that describes batching events (such as "About the JSON event protocol in HTTP Event Collector") does not mention this limit (at least, I can't find any such mention). I think it should.

0 Karma

Splunk Employee
Splunk Employee

Yes. Refer to HTTP Event Collector newly introduced with Splunk 6.3
http://dev.splunk.com/view/event-collector/SP-CAAAE6M

0 Karma

Champion

@justjosh, Yes, you can send raw event on the wire directly to a TCP. They should be in best practice format, but it should handle them just fine. In your props you will have to define how to turn the stream into an event. Like MarioM mention earlier there are to REST endpoint call recievers/simple and receivers/stream.

http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTinput#receivers.2Fstream
http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTinput#receivers.2Fsimple
http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6

Hope this helps.

Motivator

you can pass the data directly via the Splunk Rest API:

Splunk Rest API

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!