Splunk Search

Random skipped searches, scheduler slowness and UI slowness on SH.

hrawat_splunk
Splunk Employee
Splunk Employee

Encountering random skipped searches/ slow ui access.

Labels (1)
Tags (1)
0 Karma
1 Solution

hrawat_splunk
Splunk Employee
Splunk Employee

Check if auditqueue is blocked.
In Splunkd.log if you see log message `consecutive internal audit events due to blocked indexer, the event will still be in your audit log file`, all components/threads (scheduler/rest endpoints etc.) trying to generate audit log are slowing down waiting on availability of auditqueue.

Apply following workaround.
Workaround disables direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues.

1. In etc/system/local/audit.conf we can turn off audit trail direct indexing.

[auditTrail]
queueing=false

After that we have to add stanza in
etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log*]
index = _audit
source = audittrail
sourcetype = audittrail
Follow steps.
  1. Stop splunk
  2. Apply /copy above config changes.
  3. Delete all audit.log* files ( to avoid re-ingestion)
  4. Start splunk

View solution in original post

Tags (1)
0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

Check if auditqueue is blocked.
In Splunkd.log if you see log message `consecutive internal audit events due to blocked indexer, the event will still be in your audit log file`, all components/threads (scheduler/rest endpoints etc.) trying to generate audit log are slowing down waiting on availability of auditqueue.

Apply following workaround.
Workaround disables direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues.

1. In etc/system/local/audit.conf we can turn off audit trail direct indexing.

[auditTrail]
queueing=false

After that we have to add stanza in
etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log*]
index = _audit
source = audittrail
sourcetype = audittrail
Follow steps.
  1. Stop splunk
  2. Apply /copy above config changes.
  3. Delete all audit.log* files ( to avoid re-ingestion)
  4. Start splunk
Tags (1)
0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...