I have a challenge with a Twitter feed, that I have successfully integrated to the REST API Modular Input.
I need as many Tweets as possible, limited by my ability to Index them.
Presently I can't throttle back the Twitter API for my request below 52 events per second which is still too many for our needs and above our Splunk indexing license.
I would like to find a way to be able to throttle back a universal forwarder or throttle the Modular input itself.
I have two approaches in mind:
I would try option 2 above first.
There will be no issues with Twitter because by the time the received data arrives in the Universal forwarder where the limiting will be applied , the data has already been pulled down from Twitter.
Twitter -> REST Mod Input -> Response Handler -> STDOUT -> Universal Forwarder -> limits.conf -> Splunk Index