topic Which is the recommended way to input heavy load JSON data to Splunk? in Getting Data In

Which is the recommended way to input heavy load JSON data to Splunk?

wpoch — Mon, 05 Oct 2015 14:54:28 GMT

Hi everyone, I'm developing an integration with Splunk, and right now I'm using the Splunk Java SDK with the REST API and sending the events batch per documentation; the events are JSON encoded .

Now I'm considering if this is the best way to accomplish this, or should I use the TCP data inputs, or directly the Forwarders.

Any guidance will be welcome.

Thanks in advance.

Re: Which is the recommended way to input heavy load JSON data to Splunk?

alacercogitatus — Tue, 06 Oct 2015 12:12:15 GMT

While that is a perfectly good way to do it, you loose some granularity in the searches in Splunk due to the data being late coming in. The optimal way is to output to a file, and use the UF to send to Splunk. The benefits of using UF are outstanding:

In the event of network communication errors, data will be backed up on the system until the connection between UF and Indexers is restored.
Control over metadata for each file

That being said, you could also send directly to the new HTTP inputs in Splunk 6.3, but will be limiting your customer base until 6.3 is adopted widely.

Re: Which is the recommended way to input heavy load JSON data to Splunk?

wpoch — Tue, 06 Oct 2015 16:54:56 GMT

First of all, thanks for the answer.

So, in case we wan't to be able to send high throughput data over internet we keep using the REST API, and eventually migrate it to the new HTTP Event Collector. Also on controlled or on-prem environments we should use a TCP input with a previously created input on Splunk for better performance.

Finally recommend our customers, if they can install an UF use that.

We wan't to keep the UX as simple as possible, by not installing additional software (Universal Forwarder).

Kind regards,