All Apps and Add-ons

Throttling Output for the REST API Modular Input

efullman
Path Finder

I have a challenge with a Twitter feed, that I have successfully integrated to the REST API Modular Input.

I need as many Tweets as possible, limited by my ability to Index them.

Presently I can't throttle back the Twitter API for my request below 52 events per second which is still too many for our needs and above our Splunk indexing license.

I would like to find a way to be able to throttle back a universal forwarder or throttle the Modular input itself.

I have two approaches in mind:

  1. Build a response handler for the Modular Input and use the provided Response Handler field to call it. I don't have any experience with coding Python for this purpose, so any thoughts or examples are appreciated.
  2. Throttle back a Forwarder with the Modular Input installed on it by changing the "maxKBps = " in the limits.conf. Assuming I am maxing out the input at 256KBps now, I could manually cut it back to 150KBps to get underneath our licensing. I am assuming the forwarder will just discard the events that don't make it through the bandwidth. However, I am concerned that this may create a problem with Twitter who will see the client failing to keep up, and will kill the stream all together. Any experience or insight here will be helpful.
0 Karma

Damien_Dallimor
Ultra Champion

I would try option 2 above first.

There will be no issues with Twitter because by the time the received data arrives in the Universal forwarder where the limiting will be applied , the data has already been pulled down from Twitter.

Twitter -> REST Mod Input -> Response Handler -> STDOUT -> Universal Forwarder -> limits.conf -> Splunk Index

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...