Do I need TIBCO or smth like that (highload to Splunk REST API queries)

[highsplunker]

Hey guys,
I have an online connection with another web service Serv_1:
A. it sends data to MySplunk via online REST API
B. I ran a search in MySplunk to enrich the data
C. I send the enriched data back to Serv_1

Question:
What if Serv_1 sends me an enormously many API requests to MySplunk?
Do I need smth a queue manager like TIBCO between MySplunk and Serv_1?
Or my Splunk search heads cluster will do just fine?

Thanks in advance.

[jkat54]

HEC will give 500 errors when over worked. If your data is crucial to you, then yes I recommend sending data in via some method that can guarantee delivery. Such as tibco, apigee, etc. or write your own code that guarantees the delivery when any status code greater than 399 is returned.

[highsplunker]

Hi, thank you for the quick response! 🙂

I don't use HEC, and I need to reply to the Serv_1 as quickly as possible in the same API session (I guess...)...
Seems that i am forced to use some sort of smart/web service to handle the situations whe I get large number of incoming calls to MySplunk search service...

[jkat54]

That or maybe you could setup heavy forwarders to receive the data and then forward to splunk cloud.

I recommend talking to your account rep and asking for assistance.

[highsplunker]

Hi thanks, but what is"rep" ? 🙂

[jkat54]

Representative, apologies. Rep is short for representative.

Maybe you call them your account manager?

[highsplunker]

Not my case, unfortunately 🙂
Thanks

[jkat54]

I guess you can always call on a partner for consulting. It sounds like you could really use the help of someone with experience but would be best if you had them for a couple of days maybe a whole week. This might take days, weeks, or months to sort out here question by question.

You're always welcome to post as many questions here as you need though!

Cheers and hope we can help!

[nickhills · Accepted solution]

A.)
How are you sending data to Splunk - do you mean via the Http Event Collector (HEC)?
If so, the HEC is built to handle a large number of simultaneous requests, however you will need to monitor your deployment and may consider load balancing the incoming data across a 'pool' of HECs.

B.)
No Problem Here 🙂

C.)
There are a few ways to do this, but it probably depends on your exact use case.
Option 1.
Build an 'alert' which runs every x minutes to processes your data in Splunk and uses an alert action to send the data via a script back to your web service. (This is probably the approach I would take)

Option 2.
Run a saved search every x minutes which processes your data.
Create a process either on your webservice, or another host to use the Splunk Rest API and request the job results.
Transform your results as required and submit them to your webservice.

You should not need any other software (other than your custom alert/scripts) to process the data.
Whether your SHC "will do just fine" is dependent on your overall architecture and Splunk performance, but if your Splunk environment is in good health, I would think you should be ok.

[highsplunker]

@nickhillscpl thanks a lot!

[highsplunker]

Hi thanks for the quick response! 🙂

I don't use HEC, and I need to reply to the Serv_1 as quickly as possible in the same API session (I guess...).

Saved searches by schedule not applicable in my use case...