Splunk Search

How do I troubleshoot why Splunk has stopped indexing data and searching "index=_internal" produces no results?

Federica_92
Communicator

Hi everyone, I have a big issue.

Since Friday, my single node Splunk instance stopped indexing data. I was in the process of deleting and removing old app files, and I think that I accidentally disabled a default app. Before beginning the process, I created a diag file, and I have already replaced my apps folder with the folder as it was before I started to create any mess, but unluckily, Splunk still doesn't index any data.

I'm unable to search index=_internal, there are zero logs.
Looking in splunkd.log, I can't find any errors related with this problem. The logs that Splunk is suppose to read are still being collected by syslog, but they aren't being indexed.

I don't really know what else can I do. The system folder is fine, I really checked everything.
The only difference I reported was on "server control". I'm unable to restart Splunk from the GUI and a message is there:

The Splunkweb interface has been disabled. You must restart Splunk via the command line (or services control panel).  

I already tried to run the command http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/StartSplunk here to start and stop splunkweb and the status said that Splunk is correctly running.

The only errors message in splunk.d that I can find are:

  12-20-2015 14:01:01.929 +0000 WARN  ExecProcessor - Streaming XML data: Expected tag "event", instead received "error".
   12-20-2015 14:01:01.929 +0000 WARN  ExecProcessor - Streaming XML data: Expected tag "event", instead received "message".

But I don't believe that this is related with the stopped indexing. I have S.o.S and Splunk Health check installed, and both doesn't report any issues...

Please to help me, I don't really know what else can I do.

0 Karma

MuS
SplunkTrust
SplunkTrust

Check if you enabled any forwarder app; goto $SPLUNK_HOME/etc/apps and check any app.conf inside SplunkForwarder and SplunkLightForwarder for some setting like state = enabled or state = 1. If they are enabled, disable them and restart - Happy Splunking 🙂

thirumalreddyb
Communicator

What about the same errors on search head in distributed environment?

0 Karma

MuS
SplunkTrust
SplunkTrust

Are you referring to the message about the Splunkweb interface or the Streaming XML?

0 Karma

shwesinhan
New Member

thank u so much ! it works !!

0 Karma

jplumsdaine22
Influencer

Sounds like your license expired. Is everything OK in the license tab?

0 Karma

Federica_92
Communicator

yeah, everything is ok...

0 Karma

jplumsdaine22
Influencer

Ah. Well I would get in touch with Splunk Support pronto

0 Karma

Federica_92
Communicator

I did it before, but it takes really long...I'am still waiting that they reply for a case 3 weeks old..

0 Karma

jplumsdaine22
Influencer

Hopefully someone on the forum can help you faster. It's weird you don't see anything for index=_internal. Its like your old indexes are gone.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!