Getting Data In

Why has Splunk stopped indexing all log files?

cboard
Explorer

I've recently started using Splunk and it was working fine but at some point seems to have stopped indexing any logs.

I was trying to get a forwarder working so I'm guessing I've done something in trying to get the forwarder working that I've broken the main Splunk.

I've not been able to find anything relevant, everything I've come across (through Google searches) is more of a specific log not being indexed but, in my case, it seems to that every log has stopped.

From the search, I've done index=* and it only comes back with data from the 19th but I know the logs have been updated for today.

Where can I look for any problems?

Thanks

0 Karma

mthq
Engager

I seem to have a similar issue, running a standalone environment for 3 days - first two had events indexed but today I have "No results found." Monitoring single file - /var/log/mhn/mhn-splunk.log

This is college project and I seem to got stuck here. When checking splunkd.log I see:

0-21-2018 16:55:59.240 +0000 ERROR JsonLineBreaker - JSON StreamId:201389110879379108 had parsing error:Unexpected character: '-' - data_source="/var/log/mhn/mhn-splunk.log", data_host="ubuntu-s-2vcpu-4gb-ServerLondon-01", data_sourcetype="MHN"

The source file keeps getting populated.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You should post a new question.

---
If this reply helps you, Karma would be appreciated.
0 Karma

skoelpin
SplunkTrust
SplunkTrust

You should first look in splunkd to see if there's any errors. You could do this by looking at the file directly under $SPLUNK_HOME/var/log/splunk/splunkd.log or from Splunk Web by putting this in the search bar index=_internal sourcetype=splunkd error

What does your Splunk environment look like? Are you in a distributed environment or standalone system? What changes did you make to the Splunk forwarder?

0 Karma

adonio
Ultra Champion
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Where Innovation Takes Flight: The Splunk4Aviation Flight Sim Lands at .conf26

If you hear someone at .conf26 shouting "gear down, GEAR DOWN" across the show floor, you have found us.  The ...

Turn Cisco Telemetry Into Action with Cisco Data Fabric, powered by the Splunk ...

The surge in machine data is already hitting enterprise budgets, and the agentic era will only intensify it. ...

Persistent Queue at TcpOut — One of Splunk's Most Practical Features

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in ...