Getting Data In

Splunk Universal Forwader constantly crashes with "Crashing thread: indexerPipe".

dshakespeare_sp
Splunk Employee
Splunk Employee

Splunk Universal Forwader constantly crashes with "Crashing thread: indexerPipe".
splunkd.log shows:

WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false 

nclancy_splunk
Splunk Employee
Splunk Employee

I downvoted this post because wrong one vote for sorry

0 Karma

dshakespeare_sp
Splunk Employee
Splunk Employee

The interesting thing that the crashing thread is indexerPipe. The indexerPipe should be disabled on Universal Forwarder as it does not Index data.

Looking at splunkd.log we see several errors like:

WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false

This suggests that the indexPipeline is enabled and the Universal Forwarder is trying to find index components which do not exist.

On a Universal Forwarder there are several pipe lines that should be disabled. These are configured in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/default-mode.conf
which contains stanzas like

#This file turns off pipelines and processors.
#This is the default configuration

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= indexer, indexandforward, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor

[pipeline:distributedSearch]
disabled = true

[pipeline:fifo]
disabled = true

[pipeline:merging]
disabled = true

[pipeline:typing]
disabled = true 

The issue was caused by the fact the SplunkUniversalForwarder app had been deleted from $SPLUNK_HOME/etc/apps, leaving the indexing pipeline enabled.

It is not recommended to delete / change the default apps installed by default in $SPLUNK_HOME/etc/apps (introspection_generator_addon search, SplunkUniversalForwarder, learned and splunk_httpinput) or manage these apps via Splunk Deployment Server (Forwarder Managerment)

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...