Hi,
After some days the Splunk server stop receiving input. The forwarders are not changed, but I did some changes on splunk server (can't remember what I did). Also know that the firewall does not cause of the problem. On server Splunk server we have also configured Splunk Uniiversal forwarder. So same server include both Splunk Enterprise + Splunk Universal forwarder.
Not sure, but I think it's some trouble with indexer since they cannot receive inputs. Have also verified that environment variables is ok. Also changed file permission on all filres/directories below Splunk_HOME. So it should be fine
On Splunk Universal clients (on clients), splunkd.log says that TcpOutProc is connected to Splunk Server. It also says that the Splunk server LISTEN to *:9997.
> ss -tnlup
tcp LISTEN 0 128 *:9997 *:* users(("splunkd",pid=170257,fd=41))
Assume telemytry data is sent to Splunkserver, but they are not indexed. One more information:
On Splunk server: Settings - Data - Indexes I can see that
_audit SplunkLighForwarder $SPLUNK_DB/audit/db status says disabled
_internal SplunkLighForwarder $SPLUNK_DB/_internal/db status says disabled
_introspection SplunkLighForwarder $SPLUNK_DB/_introspection/db status says disabled
_telemetry SplunkLighForwarder SPLUNK_DB/_telemetry/db status says disabled
history SplunkLighForwarder SPLUNK_DB/history/db status says disabled
main SplunkLighForwarder PLUNK_DB/history /default/db status says disabled
Assume it has something to do with wrong settings on Splunk server. Hope soemone out there can give me some usefull tips/hints. So we can use splunk again as normal.
Rgds
Geir J. H
