I have an event generator that simulates five servers running uberAgent. Data is sent to Splunk via the REST API. When I start the event generator, everything is fine. But while it keeps running, the index lag keeps increasing. In other words: it takes longer and longer for the events to show up in a search.
I am seeing the REST API calls as they are made in splunkd_access.log. Example:
Possible. The REST API is rarely used for data input, and so has not been well-tested by years of field use over thousands of installations like the network, file, or program inputs. You might be better off sending data to a specified TCP port instead.