Throttling Output for the REST API Modular Input

[efullman]

I have a challenge with a Twitter feed, that I have successfully integrated to the REST API Modular Input.

I need as many Tweets as possible, limited by my ability to Index them.

Presently I can't throttle back the Twitter API for my request below 52 events per second which is still too many for our needs and above our Splunk indexing license.

I would like to find a way to be able to throttle back a universal forwarder or throttle the Modular input itself.

I have two approaches in mind:

1. Build a response handler for the Modular Input and use the provided Response Handler field to call it. I don't have any experience with coding Python for this purpose, so any thoughts or examples are appreciated.
2. Throttle back a Forwarder with the Modular Input installed on it by changing the "maxKBps = " in the limits.conf. Assuming I am maxing out the input at 256KBps now, I could manually cut it back to 150KBps to get underneath our licensing. I am assuming the forwarder will just discard the events that don't make it through the bandwidth. However, I am concerned that this may create a problem with Twitter who will see the client failing to keep up, and will kill the stream all together. Any experience or insight here will be helpful.

[Damien_Dallimor]

I would try option 2 above first.

There will be no issues with Twitter because by the time the received data arrives in the Universal forwarder where the limiting will be applied , the data has already been pulled down from Twitter.

Twitter -> REST Mod Input -> Response Handler -> STDOUT -> Universal Forwarder -> limits.conf -> Splunk Index