- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How do I troubleshoot why Splunk has stopped indexing data and searching "index=_internal" produces no results?
Hi everyone, I have a big issue.
Since Friday, my single node Splunk instance stopped indexing data. I was in the process of deleting and removing old app files, and I think that I accidentally disabled a default app. Before beginning the process, I created a diag file, and I have already replaced my apps folder with the folder as it was before I started to create any mess, but unluckily, Splunk still doesn't index any data.
I'm unable to search index=_internal, there are zero logs.
Looking in splunkd.log, I can't find any errors related with this problem. The logs that Splunk is suppose to read are still being collected by syslog, but they aren't being indexed.
I don't really know what else can I do. The system folder is fine, I really checked everything.
The only difference I reported was on "server control". I'm unable to restart Splunk from the GUI and a message is there:
The Splunkweb interface has been disabled. You must restart Splunk via the command line (or services control panel).
I already tried to run the command http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/StartSplunk here to start and stop splunkweb and the status said that Splunk is correctly running.
The only errors message in splunk.d that I can find are:
12-20-2015 14:01:01.929 +0000 WARN ExecProcessor - Streaming XML data: Expected tag "event", instead received "error".
12-20-2015 14:01:01.929 +0000 WARN ExecProcessor - Streaming XML data: Expected tag "event", instead received "message".
But I don't believe that this is related with the stopped indexing. I have S.o.S and Splunk Health check installed, and both doesn't report any issues...
Please to help me, I don't really know what else can I do.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Check if you enabled any forwarder app; goto $SPLUNK_HOME/etc/apps and check any app.conf inside SplunkForwarder and SplunkLightForwarder for some setting like state = enabled or state = 1. If they are enabled, disable them and restart - Happy Splunking 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What about the same errors on search head in distributed environment?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Are you referring to the message about the Splunkweb interface or the Streaming XML?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thank u so much ! it works !!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sounds like your license expired. Is everything OK in the license tab?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yeah, everything is ok...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ah. Well I would get in touch with Splunk Support pronto
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I did it before, but it takes really long...I'am still waiting that they reply for a case 3 weeks old..
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hopefully someone on the forum can help you faster. It's weird you don't see anything for index=_internal. Its like your old indexes are gone.
