Getting Data In

Sending input data over HTTP

justjosh
Explorer

Does Splunk support receiving a continual stream of input via an HTTP POST?

The reason I ask is the web server logs I want to index in Splunk is not accessible from the Splunk Server. I already have a tool which can relay server log events over HTTP so I think this could work.

Tags (2)
0 Karma

Graham_Hanningt
Builder

Re:

Does Splunk support receiving a continual stream of input via an HTTP POST?

No.

Not a continual (endless) stream.

You can "batch" (send multiple) events in a single HTTP POST, but there is a maximum limit to the size of an HTTP request.

The limit is set by max_content_length in limits.conf. The default value, as of Splunk 6.4, is 1000000 bytes (~ 1 MB).

Exceeding that limit results in the HTTP response error code 413 (request entity too large).

The Splunk documentation that describes batching events (such as "About the JSON event protocol in HTTP Event Collector") does not mention this limit (at least, I can't find any such mention). I think it should.

0 Karma

rarsan_splunk
Splunk Employee
Splunk Employee

Yes. Refer to HTTP Event Collector newly introduced with Splunk 6.3
http://dev.splunk.com/view/event-collector/SP-CAAAE6M

0 Karma

bmacias84
Champion

@justjosh, Yes, you can send raw event on the wire directly to a TCP. They should be in best practice format, but it should handle them just fine. In your props you will have to define how to turn the stream into an event. Like MarioM mention earlier there are to REST endpoint call recievers/simple and receivers/stream.

http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTinput#receivers.2Fstream
http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTinput#receivers.2Fsimple
http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6

Hope this helps.

MarioM
Motivator

you can pass the data directly via the Splunk Rest API:

Splunk Rest API

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...