Random skipped searches, scheduler slowness and UI slowness on SH.

[hrawat_splunk]

Encountering random skipped searches/ slow ui access.

[hrawat_splunk · Accepted solution]

Check if auditqueue is blocked.
In Splunkd.log if you see log message `consecutive internal audit events due to blocked indexer, the event will still be in your audit log file`, all components/threads (scheduler/rest endpoints etc.) trying to generate audit log are slowing down waiting on availability of auditqueue.

Apply following workaround.
Workaround disables direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues.

1. In etc/system/local/audit.conf we can turn off audit trail direct indexing.

[auditTrail]
queueing=false

After that we have to add stanza in
etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log*]
index = _audit
source = audittrail
sourcetype = audittrail

Follow steps.

1. Stop splunk
2. Apply /copy above config changes.
3. Delete all audit.log* files ( to avoid re-ingestion)
4. Start splunk