splunkd won't start

[lematyke]

The management interface ran fine until I restarted to install Universal forwarder, now splunkd will not start. Universal forwarder installed directly from the GUI. Fortunately this is a vm, so I’ve restored snapshot to just before splunk install. Unfortunately this happens each time – here’s the sequence.

• Install splunk as root using dpkg –i splunk-4.2-96430-linux-2.6-amd64.deb
• Start splunk - /opt/splunk/bin/splunk start, get the typical successful start dialogue
• Login to the management console, configure collecting data for the splunk server
• Go to manage apps, and enable universal forwarder, it then prompts to restart the server
• Click the link in management console to restart
• Restart splunk in CLI on server – splunk restart, root@deb-splunk:~# /opt/splunk/bin/splunk start splunkd

Splunk> All batbelt. No tights.

Checking prerequisites... Checking mgmt port [8089]: open Checking configuration... Done. Checking index directory... Validated databases: _audit _blocksignature _internal _thefishbucket history main summary Done Success Checking conf files for typos... All preliminary checks passed.

Starting splunk server daemon (splunkd)... Done. root@deb-splunk:~#

No restart of splunkweb appears, and doing splunk status shows:

root@deb-splunk:~# /opt/splunk/bin/splunk status splunkd 1968 was not running. Removing stale pid file... done. splunkweb is not running.

Here's the crashlog:

Received fatal signal 6 (Aborted). Cause: Signal sent by PID 2119 running under UID 0. Crashing thread: MainTailingThread Registers: RIP: [0x00007FFB77221165] gsignal + 53 (/lib/libc.so.6) RDI: [0x0000000000000847] RSI: [0x000000000000085A] RBP: [0x0000000002909A68] RSP: [0x00007FFB75DE38C8] RAX: [0x0000000000000000] RBX: [0x00000000014FE8B0] RCX: [0xFFFFFFFFFFFFFFFF] RDX: [0x0000000000000006] R8: [0x00007FFB78C93037] R9: [0x2C7472617473206F] R10: [0x0000000000000008] R11: [0x0000000000000206] R12: [0x000000000290C040] R13: [0x00007FFB75DE3A60] R14: [0x0000000002891B40] R15: [0x0000000001547E80] EFL: [0x0000000000000206] TRAPNO: [0x0000000000000000] ERR: [0x0000000000000000] CSGSFS: [0x0000000000000033] OLDMASK: [0x0000000000000000]

OS: Linux Arch: x86-64

Backtrace: [0x00007FFB77223F70] abort + 384 (/lib/libc.so.6) [0x0000000000F7D068] ZN9_gnu_cxx27__verbose_terminate_handlerEv + 200 (splunkd) [0x0000000000F7CE16] ZN10_cxxabiv111__terminateEPFvvE + 6 (splunkd) [0x0000000000F7CE43] ? (splunkd) [0x0000000000F7CF43] ? (splunkd) [0x0000000000957C66] _ZN19InputProcessorKindaC2ER6Logger + 230 (splunkd) [0x0000000000669D9A] _ZN11TailWatcherC1ERK3StrP11InputStatus + 90 (splunkd) [0x000000000066A2E4] _ZN13TailingThread4mainEv + 244 (splunkd) [0x0000000000BB03B2] _ZN6Thread8callMainEPv + 66 (splunkd) [0x00007FFB788638BA] ? (/lib/libpthread.so.0) Linux / deb-splunk / 2.6.32-5-amd64 / #1 SMP Wed Jan 12 03:40:32 UTC 2011 / x86_64 Last few lines of stderr (may contain info on assertion failure, but also could be old): 2011-03-17 11:59:18.937 -0700 Interrupt signal received 2011-03-17 11:59:30.099 -0700 splunkd started (build 96430) terminate called after throwing an instance of 'PluginException' what(): Indexer failed to start, will not continue. 2011-03-17 12:03:22.591 -0700 splunkd started (build 96430) terminate called after throwing an instance of 'PluginException' what(): Indexer failed to start, will not continue. 2011-03-17 12:11:49.390 -0700 splunkd started (build 96430) terminate called after throwing an instance of 'PluginException' what(): Indexer failed to start, will not continue. 2011-03-17 12:17:50.449 -0700 splunkd started (build 96430) terminate called after throwing an instance of 'PluginException' what(): Indexer failed to start, will not continue.

/etc/debian_version: 6.0 glibc version: 2.11.2 glibc release: stable Threads running: 13 argv: [splunkd -p 8089 start splunkd] terminating...

[Vishal_Patel]

If you want to use a Universal Forwarder, that is a separate install package entirely that can be downloaded here:

http://www.splunk.com/download/universalforwarder

The universal forwarder app should never be enabled on a full Splunk 4.2 install. I will not bore you with the explanation of why do we ship with that app in the first place, suffice it to say it is not ideal.

To remedy this issue on your full install:

% rm $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local/app.conf

% splunk restart