I have the Okta Identity Cloud Add-on for Splunk installed on a heavy forwarder. The maximum log batch size is configured at 500,000, and every other limit setting (under add-on settings) is configured at the max. For inputs, it is configured to bring in log metrics, since I am interested in authentication API requests. I ran into some issues where logs would be about an hour or two behind in the afternoon of each day, since that is when the most amount of activity on our platform occurs. I ended up having to increase the typing queue and indexing queue on this heavy forwarder (in the server.conf file) in order to fix the queueing issues this box was running into. I still notice that in the afternoon it will fall behind 15 minutes to a half hour, and then by the time morning rolls around, it is caught up.
I checked the system logs in the Okta admin portal, and I am not hitting any rate limits, or even warnings when this occurs. I am wondering if I have hit the limit of either the API, or of the add-on itself. The box that runs this heavy forwarder only has about 25% of the memory used, and 25% of the CPU in use.
