topic Re: Why are indexing queues full on the search head, but nothing has been indexing? in Getting Data In

Why are indexing queues full on the search head, but nothing has been indexing?

Haybuck15 — Fri, 28 Jul 2017 17:45:39 GMT

Hey Guys,

So, I've got a weird one. According to my monitoring console, the indexing queues on my search head are all pegged at 100%, and have been for a long time. The thing is, nothing's indexing on the thing. It's forwarding internal logs to my indexers, and I'm not running any Summary indexes on it.

Is there a way to figure out what's blocking it up? It's not a huge priority beyond the fact that the system is slow compared to other search heads, and that my boss wants to figure out why it's flagging; partially for academic reasons. Only thing I really have to go on is that on the search head, it's showing my corporate_security role with read access when I check the Introspection API; something that's not on any other system including other search heads.

Note, I don't have a Search Head cluster, but I am running a clustered pair of Indexers.

Re: Why are indexing queues full on the search head, but nothing has been indexing?

adonio — Fri, 28 Jul 2017 19:33:52 GMT

any errors or warning in internal index for that particular search head?

Re: Why are indexing queues full on the search head, but nothing has been indexing?

Haybuck15 — Fri, 28 Jul 2017 19:39:52 GMT

Nope, none whatsoever. In fact, the largest index on that search head is 3 MB; and that's only because it did some self-indexing before I configured it when it was originally stood up.

Re: Why are indexing queues full on the search head, but nothing has been indexing?

adonio — Fri, 28 Jul 2017 19:41:57 GMT

the search head outputs its data
try and search;

index = _ internal host = "yourQueuedSearchHead" log_level = warn* OR log_level = error

any results

Re: Why are indexing queues full on the search head, but nothing has been indexing?

Haybuck15 — Fri, 28 Jul 2017 19:45:28 GMT

So, it looks like the search is coming back empty, however I can do the same for every other instance of Splunk in the deployment. Is it possible that the indexing queues would fill up if it can't forward its internal logs? I mean, I wouldn't think so.

Re: Why are indexing queues full on the search head, but nothing has been indexing?

adonio — Fri, 28 Jul 2017 19:50:46 GMT

wild guess here,
check available disk space on this particular search head
if theres no space, or very minimal it can prevent splunk from indexing locally and sending the data from the search heads to indexers layer

Re: Why are indexing queues full on the search head, but nothing has been indexing?

Haybuck15 — Fri, 28 Jul 2017 19:52:38 GMT

296 GB / 299 GB Free, so that's not it.

Re: Why are indexing queues full on the search head, but nothing has been indexing?

jkat54 — Fri, 28 Jul 2017 19:55:15 GMT

There's a typo in adonio's search.

index = _internal host = "yourQueuedSearchHead" log_level = warn* OR log_level = error

Re: Why are indexing queues full on the search head, but nothing has been indexing?

Haybuck15 — Fri, 28 Jul 2017 20:03:33 GMT

I typed it by hand, I didn't copy paste it, so that's not the issue. Just doing the below search pulls back nothing.

index=_* host="yourQueuedSearchHead"

Re: Why are indexing queues full on the search head, but nothing has been indexing?

esix_splunk — Fri, 28 Jul 2017 20:11:30 GMT

Do you have an outputs.conf on your SH that forwards its logs to your indexers?

Re: Why are indexing queues full on the search head, but nothing has been indexing?

Haybuck15 — Fri, 28 Jul 2017 20:13:14 GMT

Yes, yes I do. I in fact have one on every Splunk system in the environment that's not a Universal Forwarder.

Re: Why are indexing queues full on the search head, but nothing has been indexing?

sbbadri — Fri, 28 Jul 2017 20:45:00 GMT

May be you can try clearing dispatch directory or increase indexingqueue size.

Re: Why are indexing queues full on the search head, but nothing has been indexing?

jkat54 — Fri, 28 Jul 2017 21:30:43 GMT

Are you replacing "yourQueuedSearchHead" with the host name of your queued search head?

Re: Why are indexing queues full on the search head, but nothing has been indexing?

woodcock — Sun, 30 Jul 2017 04:17:05 GMT

Are ports/ACL/routes in place to allow for your Search Head to send to your indexers (9997/9998)?