Monitoring Splunk

splunk suddenly stoped working and don't now how to fix it

NimaBokhar
Path Finder

hello all
splunk has stopped working since 2 days ago with these errors

alt text

please help me
thanks in advance

Tags (1)
0 Karma
1 Solution

DavidHourani
Super Champion

Hi @NimaBokhar,

Have a look here, this contains the full description of all health check items :
https://docs.splunk.com/Documentation/Splunk/7.2.6/DMC/Aboutfeaturemonitoring
And a guide to configure the statuses you want to keep here : https://docs.splunk.com/Documentation/Splunk/7.2.6/DMC/Configurefeaturemonitoring
In your case the TCPout is red meaning that this splunk server is not able to forward its log and this therefore drags the tailreader down with it because since there is nowhere to send the logs to then new log files cant be read. If you click on any of the red elements you should get logs pointing at what destinations cant be forwarded to and you can take the troubleshooting off from there.

Cheers,
David

View solution in original post

DavidHourani
Super Champion

Hi @NimaBokhar,

Have a look here, this contains the full description of all health check items :
https://docs.splunk.com/Documentation/Splunk/7.2.6/DMC/Aboutfeaturemonitoring
And a guide to configure the statuses you want to keep here : https://docs.splunk.com/Documentation/Splunk/7.2.6/DMC/Configurefeaturemonitoring
In your case the TCPout is red meaning that this splunk server is not able to forward its log and this therefore drags the tailreader down with it because since there is nowhere to send the logs to then new log files cant be read. If you click on any of the red elements you should get logs pointing at what destinations cant be forwarded to and you can take the troubleshooting off from there.

Cheers,
David

View solution in original post

NimaBokhar
Path Finder

thanks, David
but this is a stand-alone server and only collect logs from other devices how can it try to send logs where ever
this is the screenshot from TCPOutAutoLB:
alt text

0 Karma

DavidHourani
Super Champion

Screenshot is not showing, can you copy some of the tcpoutautolb logs ?

0 Karma

NimaBokhar
Path Finder

sorry, David fixed the screenshot but here are some of the logs:

TCPOutAutoLB-0
Root Cause: More than 70% of forwarding destinations have failed. Ensure your hosts and ports in outputs.conf are correct. Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct.
Last 50 related messages:
05-25-2019 12:18:58.909 +0430 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 8150 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
05-25-2019 12:18:48.853 +0430 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 8140 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
05-25-2019 12:18:38.798 +0430 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 8130 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

DavidHourani
Super Champion

Hi @NimaBokhar, yeah this is what I thought, please read my previous comment, seems like you're forwarding data nowhere. If this server is not forwarding data you should remove any forwarding configuration. Check for any outputs.conf in your apps that could be active. Also use : $SPLUNK_HOME$/bin/splunk btool outputs list --debug | grep default-autolb-group this will point you to the file containing the corrupt configuration.

Cheers,
David

0 Karma

NimaBokhar
Path Finder

thanks man that worked like a charm.

0 Karma

DavidHourani
Super Champion

Awesome man 😉 glad I could help

0 Karma

NimaBokhar
Path Finder

appreciate it 😉

0 Karma

DavidHourani
Super Champion

Okay, now it's working. Look in your config files on that Splunk for an outputs.conf file that contains the following output group default-autolb-group. It seems like you have log forwarding enabled but it's pointing nowhere. If it's a standalone and no where to send the logs to then you will have to get rid of this configuration.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!