I am setting up my first cluster in a lab at my house.
My setup is:
1 Cluster Manager: cm
2 Indexers: idx01, idx02
1 Search head: sh01
1 Universal Forwarder: uf01
Each is a dedicated VM with 4 GB ram and 2 CPU's assigned to it.
Each had 30 to 100 GB of disk space assigned to it with plenty of free space on each.
At this point, I have not set up a single index or begun forwarding ANY data except the default data that is generated by Splunk itself.
I wanted to make sure everything looked right, before starting to send the cluster data.
The cluster manager health status shows:
Root Cause(s): The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data. Last 50 related messages: 08-11-2019 18:07:25.311 +0000 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... 08-11-2019 18:07:14.089 +0000 INFO TailReader - Starting batchreader0 thread 08-11-2019 18:07:14.089 +0000 INFO TailReader - Registering metrics callback for: batchreader0
When I view the Data Pipeline for both Indexers (also looked at cm)
idx01 shows parsing queue 99% merging queue 99%, typing queue 99%, and Index queue at 100%.
All other machines are at 0%.
When I look at sh01 it shows idx01 and idx01 State: up, Status Healthy, HC failures None, Status enabled.
I have read a few pages on troubleshooting queues and pipelines. I believe the idx01 and idx02 are set up identically. I have no idea why idx01 displays this problem and nothing similar in idx02. Obviously, it isn't 100% identical... 🙂
check the inputs.conf on indexers and outputs.conf on non-indexers.
disabled = 0
index = false
defaultGroup = indexing_cluster
forwardedindex.filter.disable = true
indexAndForward = false
Login to the server on which you have enabled monitoring console . go to settings > monitoring console > health check .
run this script . Please note before running the health check script before configure monitoring console as distributed with required server roles. to this - inside monitoring console go to settings > General Setup > Distributed
To start off with... (Thank-You!)
After posting the question last night, I found someone with a kind of similar question who said they queue sizes by default are too small, so I enlarged them. That did empty the queues right away. Keeping in mind I am not sending any data at all, that just seemed odd.
To put things back an work with your advice, I have removed my enhanced queues settings and restarted.
I am assuming the the master node counts as a non idexing server so setting the outputs.conf there also.
On the non-indexing servers it shows: server=:9997,:9997
I currently have it set to i.p.add.ress1:9997,i.p.add.ress2:9997.
I have MC enabled on anything with a gui. I was unaware it shouldn't be that way.
I'll use the cluster manager for monitoring console in your reply.
I noticed for the first time it says 'standalone'
odd, it says the cm is a search head.
It took me about 45 minutes to figure out how to edit the roles. Apparently if the browser isn't open far enough to the right, you can't see the 'Action' edit link. Good learning experience.
I tried to remove all roles by editing the 'Action' link to only be a Cluster Master, and it replies with a 'something went wrong, please try again later. I also tries editing assets.csv and rebooting the machine. No luck.
I'll just go ahead as it is. (I am using a free 60 day trial, so maybe that's the problem)
I ignored the warning and set the MC to distributed.
First I ran the health check local, and no errors, than I ran it distributed, and it replied with 18 ALL 18 N/A.
The two indexers only show the indexer server and license master I guess because everything relies on a free 60 day enterprise license.
Indexing pipeline as shown from CM is 0%.
From idx01 the Index Queue is at 2%
Fromn idx02 it is at 12%
Thank-You for your time.