Getting Data In

Splunk Universal Forwader constantly crashes with "Crashing thread: indexerPipe".

dshakespeare_sp
Splunk Employee
Splunk Employee

Splunk Universal Forwader constantly crashes with "Crashing thread: indexerPipe".
splunkd.log shows:

WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false 

nclancy_splunk
Splunk Employee
Splunk Employee

I downvoted this post because wrong one vote for sorry

0 Karma

dshakespeare_sp
Splunk Employee
Splunk Employee

The interesting thing that the crashing thread is indexerPipe. The indexerPipe should be disabled on Universal Forwarder as it does not Index data.

Looking at splunkd.log we see several errors like:

WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false

This suggests that the indexPipeline is enabled and the Universal Forwarder is trying to find index components which do not exist.

On a Universal Forwarder there are several pipe lines that should be disabled. These are configured in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/default-mode.conf
which contains stanzas like

#This file turns off pipelines and processors.
#This is the default configuration

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= indexer, indexandforward, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor

[pipeline:distributedSearch]
disabled = true

[pipeline:fifo]
disabled = true

[pipeline:merging]
disabled = true

[pipeline:typing]
disabled = true 

The issue was caused by the fact the SplunkUniversalForwarder app had been deleted from $SPLUNK_HOME/etc/apps, leaving the indexing pipeline enabled.

It is not recommended to delete / change the default apps installed by default in $SPLUNK_HOME/etc/apps (introspection_generator_addon search, SplunkUniversalForwarder, learned and splunk_httpinput) or manage these apps via Splunk Deployment Server (Forwarder Managerment)

Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...