After some days the Splunk server stop receiving input.  The forwarders are not changed, but I did some changes on splunk server (can't remember what I did).  Also know that the firewall does not cause of the problem. On server Splunk server we have also configured Splunk Uniiversal forwarder. So same server include both Splunk Enterprise + Splunk Universal forwarder. 


Not sure, but I think it's some trouble with indexer since they cannot receive inputs. Have also  verified that environment variables is ok.  Also changed file permission on all filres/directories below Splunk_HOME.   So it should be fine


On Splunk Universal clients (on clients),  splunkd.log says that TcpOutProc is connected to Splunk Server. It also says that the Splunk server LISTEN to *:9997. 

> ss -tnlup

tcp LISTEN 0 128 *:9997  *:* users(("splunkd",pid=170257,fd=41))


Assume telemytry data is sent to Splunkserver, but they are not indexed. One more information: 


On Splunk server: Settings - Data - Indexes  I can see that 

_audit SplunkLighForwarder $SPLUNK_DB/audit/db status says  disabled

_internal SplunkLighForwarder $SPLUNK_DB/_internal/db status says  disabled

_introspection SplunkLighForwarder $SPLUNK_DB/_introspection/db status says  disabled

_telemetry  SplunkLighForwarder SPLUNK_DB/_telemetry/db status says  disabled

history SplunkLighForwarder SPLUNK_DB/history/db status says  disabled

main  SplunkLighForwarder PLUNK_DB/history /default/db status says  disabled


Assume it has something to do with wrong settings on Splunk server.  Hope soemone out there can give me some usefull tips/hints. So we can use splunk again as normal. 



Geir J. H

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...