Getting Data In

Splunk Universal Forwader constantly crashes with "Crashing thread: indexerPipe".

Splunk Employee
Splunk Employee

Splunk Universal Forwader constantly crashes with "Crashing thread: indexerPipe".
splunkd.log shows:

WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false 

Splunk Employee
Splunk Employee

I downvoted this post because wrong one vote for sorry

0 Karma

Splunk Employee
Splunk Employee

The interesting thing that the crashing thread is indexerPipe. The indexerPipe should be disabled on Universal Forwarder as it does not Index data.

Looking at splunkd.log we see several errors like:

WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false

This suggests that the indexPipeline is enabled and the Universal Forwarder is trying to find index components which do not exist.

On a Universal Forwarder there are several pipe lines that should be disabled. These are configured in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/default-mode.conf
which contains stanzas like

#This file turns off pipelines and processors.
#This is the default configuration

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= indexer, indexandforward, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor

[pipeline:distributedSearch]
disabled = true

[pipeline:fifo]
disabled = true

[pipeline:merging]
disabled = true

[pipeline:typing]
disabled = true 

The issue was caused by the fact the SplunkUniversalForwarder app had been deleted from $SPLUNK_HOME/etc/apps, leaving the indexing pipeline enabled.

It is not recommended to delete / change the default apps installed by default in $SPLUNKHOME/etc/apps (introspectiongeneratoraddon search, SplunkUniversalForwarder, learned and splunkhttpinput) or manage these apps via Splunk Deployment Server (Forwarder Managerment)