Getting Data In

Splunk Universal Forwader constantly crashes with "Crashing thread: indexerPipe".

dshakespeare_sp
Splunk Employee
Splunk Employee

Splunk Universal Forwader constantly crashes with "Crashing thread: indexerPipe".
splunkd.log shows:

WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false 

nclancy_splunk
Splunk Employee
Splunk Employee

I downvoted this post because wrong one vote for sorry

0 Karma

dshakespeare_sp
Splunk Employee
Splunk Employee

The interesting thing that the crashing thread is indexerPipe. The indexerPipe should be disabled on Universal Forwarder as it does not Index data.

Looking at splunkd.log we see several errors like:

WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false

This suggests that the indexPipeline is enabled and the Universal Forwarder is trying to find index components which do not exist.

On a Universal Forwarder there are several pipe lines that should be disabled. These are configured in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/default-mode.conf
which contains stanzas like

#This file turns off pipelines and processors.
#This is the default configuration

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= indexer, indexandforward, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor

[pipeline:distributedSearch]
disabled = true

[pipeline:fifo]
disabled = true

[pipeline:merging]
disabled = true

[pipeline:typing]
disabled = true 

The issue was caused by the fact the SplunkUniversalForwarder app had been deleted from $SPLUNK_HOME/etc/apps, leaving the indexing pipeline enabled.

It is not recommended to delete / change the default apps installed by default in $SPLUNK_HOME/etc/apps (introspection_generator_addon search, SplunkUniversalForwarder, learned and splunk_httpinput) or manage these apps via Splunk Deployment Server (Forwarder Managerment)

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...