Encountering random skipped searches/ slow ui access.
Check if auditqueue is blocked.
In Splunkd.log if you see log message `consecutive internal audit events due to blocked indexer, the event will still be in your audit log file`, all components/threads (scheduler/rest endpoints etc.) trying to generate audit log are slowing down waiting on availability of auditqueue.
Apply following workaround.
Workaround disables direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues.
1. In etc/system/local/audit.conf we can turn off audit trail direct indexing. [auditTrail] queueing=false After that we have to add stanza in etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log [monitor://$SPLUNK_HOME/var/log/splunk/audit.log*] index = _audit source = audittrail sourcetype = audittrail
Check if auditqueue is blocked.
In Splunkd.log if you see log message `consecutive internal audit events due to blocked indexer, the event will still be in your audit log file`, all components/threads (scheduler/rest endpoints etc.) trying to generate audit log are slowing down waiting on availability of auditqueue.
Apply following workaround.
Workaround disables direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues.
1. In etc/system/local/audit.conf we can turn off audit trail direct indexing. [auditTrail] queueing=false After that we have to add stanza in etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log [monitor://$SPLUNK_HOME/var/log/splunk/audit.log*] index = _audit source = audittrail sourcetype = audittrail