Splunk Search

Blocked auditqueue causing random skipped searches and scheduler slowness on SH/SHC. Slow UI.

hrawat
Splunk Employee
Splunk Employee

Blocked auditqueue can cause random skipped searches, scheduler slowness on SH/SHC and slow UI.

Tags (1)
0 Karma
1 Solution

hrawat
Splunk Employee
Splunk Employee

With every new major release of splunk, more and more components are adding audit logs. Volume of audit logs has increased significantly. UI/Search Scheduler/Search Dispatcher etc generate audit logs.
If auditqueue is full, then all these components are getting serialized to insert audit event into auditqueue. Resulting in skipped searches/ slow UI login etc.

To mitigate the problem apply following workaround by using file monitoring instead of in-memory indexing.

Workaround
Disable direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues.

Steps.
1. In etc/system/local/audit.conf ( or any audit.conf you like) we can turn off audit trail direct indexing.

[auditTrail]
queueing=false

After that we have to add stanza in etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log*]
index = _audit
source = audittrail
sourcetype = audittrail

2. Stop splunk

3. Delete all audit.log* files ( to avoid re-ingestion). This step is optional if you don't care about duplicate audit events.

4. Start splunk


 

View solution in original post

hrawat
Splunk Employee
Splunk Employee

With every new major release of splunk, more and more components are adding audit logs. Volume of audit logs has increased significantly. UI/Search Scheduler/Search Dispatcher etc generate audit logs.
If auditqueue is full, then all these components are getting serialized to insert audit event into auditqueue. Resulting in skipped searches/ slow UI login etc.

To mitigate the problem apply following workaround by using file monitoring instead of in-memory indexing.

Workaround
Disable direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues.

Steps.
1. In etc/system/local/audit.conf ( or any audit.conf you like) we can turn off audit trail direct indexing.

[auditTrail]
queueing=false

After that we have to add stanza in etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log*]
index = _audit
source = audittrail
sourcetype = audittrail

2. Stop splunk

3. Delete all audit.log* files ( to avoid re-ingestion). This step is optional if you don't care about duplicate audit events.

4. Start splunk


 

Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Your Voice Matters! Help Us Shape the New Splunk Lantern Experience

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Building Momentum: Splunk Developer Program at .conf25

At Splunk, developers are at the heart of innovation. That’s why this year at .conf25, we officially launched ...