The interesting thing that the crashing thread is indexerPipe. The indexerPipe should be disabled on Universal Forwarder as it does not Index data.
Looking at splunkd.log we see several errors like:
WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
ERROR IndexConfig - stanza=default Required parameter=defaultDatabase not configured
FATAL IndexerService - Cannot load IndexConfig: stanza=default Required parameter=defaultDatabase not configured
INFO IndexProcessor - Initializing: readonly=false reloading=false
This suggests that the indexPipeline is enabled and the Universal Forwarder is trying to find index components which do not exist.
On a Universal Forwarder there are several pipe lines that should be disabled. These are configured in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/default-mode.conf
which contains stanzas like
#This file turns off pipelines and processors.
#This is the default configuration
#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= indexer, indexandforward, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor
[pipeline:distributedSearch]
disabled = true
[pipeline:fifo]
disabled = true
[pipeline:merging]
disabled = true
[pipeline:typing]
disabled = true
The issue was caused by the fact the SplunkUniversalForwarder app had been deleted from $SPLUNK_HOME/etc/apps, leaving the indexing pipeline enabled.
It is not recommended to delete / change the default apps installed by default in $SPLUNK_HOME/etc/apps (introspection_generator_addon search, SplunkUniversalForwarder, learned and splunk_httpinput) or manage these apps via Splunk Deployment Server (Forwarder Managerment)
