With every new major release of splunk, more and more components are adding audit logs. Volume of audit logs has increased significantly. UI/Search Scheduler/Search Dispatcher etc generate audit logs. If auditqueue is full, then all these components are getting serialized to insert audit event into auditqueue. Resulting in skipped searches/ slow UI login etc. To mitigate the problem apply following workaround by using file monitoring instead of in-memory indexing. Workaround Disable direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues. Steps. 1. In etc/system/local/audit.conf ( or any audit.conf you like) we can turn off audit trail direct indexing. [auditTrail] queueing=false After that we have to add stanza in etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log [monitor://$SPLUNK_HOME/var/log/splunk/audit.log*] index = _audit source = audittrail sourcetype = audittrail 2. Stop splunk 3. Delete all audit.log* files ( to avoid re-ingestion). This step is optional if you don't care about duplicate audit events. 4. Start splunk   ... View more

If HEC server is continuously sending 503 reply with "Server is busy", you can discard this reply. If HEC server is intermittently sending 503 reply with "Server is busy", then first understand following fields. events_processed=number of events successfully inserted into pipeline queue http_input_body_size=http POST payload size sent by HEC client HEC server knows the http POST payload size, but it does not know how many events the payload has. So after inserting events_processed events into the pipeline queue, HEC server receiver thread finds that now the pipeline queue is blocked. It still has some unknown number of events un-processed. Waits for 1 sec for the queue to have space to insert next event. If still not enough space in the queue, then HEC server drops remaining events of the payload and replies 503 "Server is busy". To mitigate this problem, always ensure the parsingqueue is atleast 10 times more than the max POST payload. In server.conf [queue=parsingQueue] maxSize = <minimum 10 x (max expected POST payload size)> ... View more

Check if auditqueue is blocked. In Splunkd.log if you see log message `consecutive internal audit events due to blocked indexer, the event will still be in your audit log file`, all components/threads (scheduler/rest endpoints etc.) trying to generate audit log are slowing down waiting on availability of auditqueue. Apply following workaround. Workaround disables direct indexing of audit events and instead fallback on file monitoring. This workaround decouples scheduler/UI threads from ingestion pipeline queues. 1. In etc/system/local/audit.conf we can turn off audit trail direct indexing. [auditTrail] queueing=false After that we have to add stanza in etc/system/local/inputs.conf( or any inputs.conf you like) to monitor audit.log [monitor://$SPLUNK_HOME/var/log/splunk/audit.log*] index = _audit source = audittrail sourcetype = audittrail Follow steps. Stop splunk Apply /copy above config changes. Delete all audit.log* files ( to avoid re-ingestion) Start splunk ... View more

It's possible third party S2S client has enabled/proxied acknowledge (useACK=true), however most of the 3rd party clients are unable to handle acknowledge received from indexers/receivers. After sometime indexer/receiver aborts(assertion failure) after detecting that S2S client is unable to process ACKs. Workaround: Turn off useACK on third party S2S client side. Turn off useACK on UF if it routes via 3rd party S2S client. Note: use Splunk INGEST ACTIONS instead of 3rd party S2S client.    ... View more