Getting Data In

Why isn't my data reaching the index?

ddrillic
Ultra Champion

We have a small farm with no access to the forwarders. The forwarders do phone home but the following returns nothing when running it against all time -

index=_internal host=<fwdhost1>* OR host=<fwdhost2>* OR host=<fwdhost3>* OR host=<fwdhost4>* OR host=<fwdhost5>*

Nothing reaches the index.

How can we debug it further?

0 Karma
1 Solution

mdsnmss
SplunkTrust
SplunkTrust

Is it a distributed or standalone deployment? If distributed, I would check if it is connecting to the indexers. You can get a phone home without connecting to the indexers since the phone home goes through the deployment server. You can use the search to look for the hosts to see if they are connecting:

index=_internal source=*metrics.log tcpin_connections

If not, it could be for a variety of reasons. Firewall preventing it, no forward-server listed on the forwarder, or monitor not working on the forwarders.

View solution in original post

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

Is it a distributed or standalone deployment? If distributed, I would check if it is connecting to the indexers. You can get a phone home without connecting to the indexers since the phone home goes through the deployment server. You can use the search to look for the hosts to see if they are connecting:

index=_internal source=*metrics.log tcpin_connections

If not, it could be for a variety of reasons. Firewall preventing it, no forward-server listed on the forwarder, or monitor not working on the forwarders.

View solution in original post

0 Karma

ddrillic
Ultra Champion

It's a distributed deployment and this query returns events for the two indexers. I don't see the forwarders listed. Should they?

You said -

-- If not, it could be for a variety of reasons.

Meaning, the forwarders apparently can't reach the indexers, right?

I ran this query on our core farm - index=_internal source=*metrics.log tcpin_connections and it doesn't show the forwarders...

I ran - index=_internal sourcetype=splunkd component=TcpOutputProc (host=<fwdhost1>* OR host=<fwdhost2>**)

Nothing came back. So, I guess no connectivity exists between the forwarders and the indexers...

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

Your forwarders would be listed as the sourceHost field. Likely as an IP rather than a hostname. If you don't see the forwarders listed it would likely mean they are not reaching the indexers.

0 Karma

ddrillic
Ultra Champion

Perfect - much appreciated.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!