I have a challenge with a Twitter feed, that I have successfully integrated to the REST API Modular Input.
I need as many Tweets as possible, limited by my ability to Index them.
Presently I can't throttle back the Twitter API for my request below 52 events per second which is still too many for our needs and above our Splunk indexing license.
I would like to find a way to be able to throttle back a universal forwarder or throttle the Modular input itself.
I have two approaches in mind:
Build a response handler for the Modular Input and use the provided Response Handler field to call it. I don't have any experience with coding Python for this purpose, so any thoughts or examples are appreciated.
Throttle back a Forwarder with the Modular Input installed on it by changing the "maxKBps = " in the limits.conf. Assuming I am maxing out the input at 256KBps now, I could manually cut it back to 150KBps to get underneath our licensing. I am assuming the forwarder will just discard the events that don't make it through the bandwidth. However, I am concerned that this may create a problem with Twitter who will see the client failing to keep up, and will kill the stream all together. Any experience or insight here will be helpful.