Splunk Search

How do I troubleshoot why Splunk has stopped indexing data and searching "index=_internal" produces no results?

Federica_92
Communicator

Hi everyone, I have a big issue.

Since Friday, my single node Splunk instance stopped indexing data. I was in the process of deleting and removing old app files, and I think that I accidentally disabled a default app. Before beginning the process, I created a diag file, and I have already replaced my apps folder with the folder as it was before I started to create any mess, but unluckily, Splunk still doesn't index any data.

I'm unable to search index=_internal, there are zero logs.
Looking in splunkd.log, I can't find any errors related with this problem. The logs that Splunk is suppose to read are still being collected by syslog, but they aren't being indexed.

I don't really know what else can I do. The system folder is fine, I really checked everything.
The only difference I reported was on "server control". I'm unable to restart Splunk from the GUI and a message is there:

The Splunkweb interface has been disabled. You must restart Splunk via the command line (or services control panel).  

I already tried to run the command http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/StartSplunk here to start and stop splunkweb and the status said that Splunk is correctly running.

The only errors message in splunk.d that I can find are:

  12-20-2015 14:01:01.929 +0000 WARN  ExecProcessor - Streaming XML data: Expected tag "event", instead received "error".
   12-20-2015 14:01:01.929 +0000 WARN  ExecProcessor - Streaming XML data: Expected tag "event", instead received "message".

But I don't believe that this is related with the stopped indexing. I have S.o.S and Splunk Health check installed, and both doesn't report any issues...

Please to help me, I don't really know what else can I do.

0 Karma

MuS
SplunkTrust
SplunkTrust

Check if you enabled any forwarder app; goto $SPLUNK_HOME/etc/apps and check any app.conf inside SplunkForwarder and SplunkLightForwarder for some setting like state = enabled or state = 1. If they are enabled, disable them and restart - Happy Splunking 🙂

thirumalreddyb
Communicator

What about the same errors on search head in distributed environment?

0 Karma

MuS
SplunkTrust
SplunkTrust

Are you referring to the message about the Splunkweb interface or the Streaming XML?

0 Karma

shwesinhan
New Member

thank u so much ! it works !!

0 Karma

jplumsdaine22
Influencer

Sounds like your license expired. Is everything OK in the license tab?

0 Karma

Federica_92
Communicator

yeah, everything is ok...

0 Karma

jplumsdaine22
Influencer

Ah. Well I would get in touch with Splunk Support pronto

0 Karma

Federica_92
Communicator

I did it before, but it takes really long...I'am still waiting that they reply for a case 3 weeks old..

0 Karma

jplumsdaine22
Influencer

Hopefully someone on the forum can help you faster. It's weird you don't see anything for index=_internal. Its like your old indexes are gone.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...