Getting Data In

How can I monitor Splunk instances (deployment, deployer, seach head, cluster master, etc.)?

bryanthomas24vs
Explorer

I'm new to setting up clusters and I assumed that the splunk instances (deployment, deployer, seach head, cluster master, etc) would naturally forward their own splunk logs ($SPLUNK_HOME/var/log/splunk/*).

Are they and I am missing them? host=cluster-master index=* no results...
Is there an app, best practice, or some documentation regarding this?
I can't find what I am looking for in the monitoring console, which is:

Failed to register with cluster master
reason: failed method=POST
expected_response_code=2xx
actual_response_code=500
status_line="Internal Server Error"

Everything seems to be working fine...

0 Karma

paulstout
Path Finder

Have you set up outputs.conf on your components? Splunk requires outputs.conf to be configured -- same as the universal forwarders -- before these components will send their logs to your index cluster.

You can find more information in this documentation link:

https://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/Forwardsearchheaddata

bryanthomas24vs
Explorer

Short answer: yes.
Long answer: (I didn't configure this bit) On the cluster master, in /opt/splunk/etc/apps/cluster_forwarder_outputs/local/outputs.conf it has:
[tcpout]
defaultGroup = primary_indexers
maxQueueSize = 7MB
useACK = true
forceTimebasedAutoLB = true
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
[tcpout:primary_indexers]
server = indxer01.fqdn:9997, indxer02.fqdn:9997
autoLB = true

I can see the instances (5) in the CM under overview & resources, but nowhere else.

0 Karma

bryanthomas24vs
Explorer

but I also found the following in another outputs.conf, which I am guessing is the cause of the problem:
[indexAndForward]
index = false
[tcpout]
forwardedindex.filter.disable = true
indexAndForward = false

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...