All Apps and Add-ons

Throttling Output for the REST API Modular Input

efullman
Path Finder

I have a challenge with a Twitter feed, that I have successfully integrated to the REST API Modular Input.

I need as many Tweets as possible, limited by my ability to Index them.

Presently I can't throttle back the Twitter API for my request below 52 events per second which is still too many for our needs and above our Splunk indexing license.

I would like to find a way to be able to throttle back a universal forwarder or throttle the Modular input itself.

I have two approaches in mind:

  1. Build a response handler for the Modular Input and use the provided Response Handler field to call it. I don't have any experience with coding Python for this purpose, so any thoughts or examples are appreciated.
  2. Throttle back a Forwarder with the Modular Input installed on it by changing the "maxKBps = " in the limits.conf. Assuming I am maxing out the input at 256KBps now, I could manually cut it back to 150KBps to get underneath our licensing. I am assuming the forwarder will just discard the events that don't make it through the bandwidth. However, I am concerned that this may create a problem with Twitter who will see the client failing to keep up, and will kill the stream all together. Any experience or insight here will be helpful.
0 Karma

Damien_Dallimor
Ultra Champion

I would try option 2 above first.

There will be no issues with Twitter because by the time the received data arrives in the Universal forwarder where the limiting will be applied , the data has already been pulled down from Twitter.

Twitter -> REST Mod Input -> Response Handler -> STDOUT -> Universal Forwarder -> limits.conf -> Splunk Index

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...