Building for the Splunk Platform

JAVA SDK is taking too long time to index data to splunk

royalhrk
Loves-to-Learn Lots

Hi,

We are using custom java program to index data to splunk using Java SDK (1.6.5.0).  We have huge volume to data to index on daily bases around  aprox.100000 documents.

We used TCP method but it requires a port to open and bind with a specific source to index data. We receive data as bulk (several records) and process it and index each record seperately. to index data as unique event we need to connect TCP port each time and close it to index data. This way it is working very slow (might be  due to connecting and closing TCP port for each record).  We tried the com.splunk.Index API (submit()/ attach()), but index rate is very slow.

We have java parsers that are doing some business specific stuff with data, so dont want to go for forwards to rewrite the data parsing logic.

Can anyone help here and suggest the best practice/approach to index, using java SDK (considering high data velicity).

Thanks.

Labels (2)
Tags (2)
0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! 👏 Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...