Hi everyone, I have a big issue.
Since Friday, my single node Splunk instance stopped indexing data. I was in the process of deleting and removing old app files, and I think that I accidentally disabled a default app. Before beginning the process, I created a diag file, and I have already replaced my apps folder with the folder as it was before I started to create any mess, but unluckily, Splunk still doesn't index any data.
I'm unable to search
index=_internal, there are zero logs.
Looking in splunkd.log, I can't find any errors related with this problem. The logs that Splunk is suppose to read are still being collected by syslog, but they aren't being indexed.
I don't really know what else can I do. The system folder is fine, I really checked everything.
The only difference I reported was on "server control". I'm unable to restart Splunk from the GUI and a message is there:
The Splunkweb interface has been disabled. You must restart Splunk via the command line (or services control panel).
I already tried to run the command http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/StartSplunk here to start and stop splunkweb and the status said that Splunk is correctly running.
The only errors message in splunk.d that I can find are:
12-20-2015 14:01:01.929 +0000 WARN ExecProcessor - Streaming XML data: Expected tag "event", instead received "error". 12-20-2015 14:01:01.929 +0000 WARN ExecProcessor - Streaming XML data: Expected tag "event", instead received "message".
But I don't believe that this is related with the stopped indexing. I have S.o.S and Splunk Health check installed, and both doesn't report any issues...
Please to help me, I don't really know what else can I do.
Check if you enabled any forwarder app; goto $SPLUNK_HOME/etc/apps and check any
SplunkLightForwarder for some setting like
state = enabled or
state = 1. If they are enabled, disable them and restart - Happy Splunking 🙂