Why isn't my data reaching the index?

[ddrillic]

We have a small farm with no access to the forwarders. The forwarders do phone home but the following returns nothing when running it against all time -

index=_internal host=<fwdhost1>* OR host=<fwdhost2>* OR host=<fwdhost3>* OR host=<fwdhost4>* OR host=<fwdhost5>*

Nothing reaches the index.

How can we debug it further?

[mdsnmss · Accepted solution]

Is it a distributed or standalone deployment? If distributed, I would check if it is connecting to the indexers. You can get a phone home without connecting to the indexers since the phone home goes through the deployment server. You can use the search to look for the hosts to see if they are connecting:

index=_internal source=*metrics.log tcpin_connections

If not, it could be for a variety of reasons. Firewall preventing it, no forward-server listed on the forwarder, or monitor not working on the forwarders.

[ddrillic]

It's a distributed deployment and this query returns events for the two indexers. I don't see the forwarders listed. Should they?

You said -

-- If not, it could be for a variety of reasons.

Meaning, the forwarders apparently can't reach the indexers, right?

I ran this query on our core farm - index=_internal source=*metrics.log tcpin_connections and it doesn't show the forwarders...

I ran - index=_internal sourcetype=splunkd component=TcpOutputProc (host=<fwdhost1>* OR host=<fwdhost2>**)

Nothing came back. So, I guess no connectivity exists between the forwarders and the indexers...

[mdsnmss]

Your forwarders would be listed as the sourceHost field. Likely as an IP rather than a hostname. If you don't see the forwarders listed it would likely mean they are not reaching the indexers.

[ddrillic]

Perfect - much appreciated.