Splunk is not indexing anymore

[fedayn05]

Hello Team,

I hope you are doing well. 

Recently i am going through a critical issue on my splunk entreprise. I used to receive logs from switches and firewalls and everything was fine.

Until yesterday, splunk stopped indexing everything, at first i thought the firewalls stopped sending logs , but it was not the case , i even launched the tcpdump command on the VM hosting splunk , and i see that the logs arrives , but when i search on splunk , i cannot find anything.

If you have any idea please or you could suggest any ideas , I would appreciate your help.

I m using splunk entrprise 10.0.0 hosted on ubuntu 22.04. Also i do not have any shortage when it comes to resources.

Thank you for your time

[fedayn05 · Accepted solution]

Hello Team ,

Thank you all for your replies. Actually after deep investigations it turned out it was a mistake made by one of our team.

To put you in the context, we have had an issue in Splunk , the Forwarder agents won't show up in the agent mangement section, son one of the Team looked this up and found a similar case and it was resolved by adding those lines on outputs.conf : 

[indexAndForward]

index = true

selectiveIndexing = true

and this the issue all indexes stopped indexing. by removing those lines everything got back to work noemally.

Thank you again for you help.

[isoutamo]

Are you sure that splunk has stopped to indexing or is it stopped to answer your queries? The 2nd option is more possible than 1st one!
So how you have made conclusion that it has stopped ingestion?

[livehybrid]

Hi @fedayn05 

Can you provide a little more info on how you are receiving this data into Splunk?

When you look at https://yoursplunkinstance/en-US/manager/bowl/data/indexes do you see the index that you are expecting to see logs for? When does it say the most recent event is from?

Can you check Splunk is listening on the relevant port (e.g. can you netcat to it or see it open in some other way?)

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

It's very hard to tell with such limited info.

1. Check if all your components are up and running.

2. If you're saying that "logs arrive", since you're talking about network equipment, do you mean that you see just UDP frames or TCP streams? Remember that UDP can appear on the wire but still not get delivered to the process even if the process is listening. Any changes on the boxes network-wise? Someone fiddled with the local firewall rules?

3. Is Splunk indexing _any_ data? Do you have only problem with the network equipment logs or everything (including Splunk's _internal)?

4. How did you verify that it stopped indexing? Maybe something changed regarding time settings - that could cause Splunk to parse time wrongly and index it into wrong point in time (effectively not showing it when you're searching for "last 15 minutes" or similar). Try running an All Time (realtime) search across your indexes and look if anything shows up. (that's pretty much the only case I'd advise anyone to run a realtime search).

5. Did you check indexes metadata (both on Monitoring Console level as well as physically on bucket directories)?