Getting Data In

Why are only 10 of my 16 forwarders showing up in my receiver/indexer?

pwhitebe
Engager

Greetings,

I have set up 17 micro AWS boxes, One running a splunk 6.2.0 indexer, 8 with databases (8 mongo and 4 mongo and neo4j), 8 with Node.JS, and set them up with splunk 6.2.0 heavy forwarders monitoring relevant files and forwarding to the splunk receiver/indexer. The problem is but only 10 of the forwarding instances ever show up in the indexer.

The receiver seems to only see the most-recent 10 of them in the data summary. 6 of the newest boxes seem to just not show up in the data summary on the main receiver/indexer.

These 6 boxes appear to be configured properly: They have monitors on the relevant files when I 'splunk list monitor'. They also show the receiver as an "active forward" when I 'splunk list forward-server'. As far as I can tell they are set up the same as the other 10 boxes that work.

So where is the block/issue? Is there some 10-forwarder limit I am hitting? Is there a concurrent search limit manifesting as a 10-forwarder limit? Do I need to do a split across load balancers if the receiver and indexer are on the same machine?

Thanks in advance for any assistance.

Tags (3)
0 Karma
1 Solution

woodcock
Esteemed Legend

There is no 10-forwarder limit.

How are you deploying your configurations (I assume manually)? For the servers where the configuration are working, where did you deploy your configuration files? Was it in $SPLUNK_HOME/etc/system/local/ (hopefully it was in $SPLUNK_HOME/etc/system/MyApp/default/). In any case, wherever that was, just copy that subdirectory as-is from a working server to a non-working server. In particular, I suspect you have a mismatch with outputs.conf. Also pay careful attention to server.conf which may include configurations that are supposed to be host/server specific and should*NOT* be copied without being adjusted. It might help to do a search like this from your search head (lets' say your hostname is "myhost" and your IP address is "172.0.0.1"):

index=_internal host=hostname OR host=172.0.0.1 OR hostname OR 172.0.0.1

Also, login to a broken forwarder with your browser and search locally for problems like this:

index=_internal WARN or ERR*

View solution in original post

woodcock
Esteemed Legend

There is no 10-forwarder limit.

How are you deploying your configurations (I assume manually)? For the servers where the configuration are working, where did you deploy your configuration files? Was it in $SPLUNK_HOME/etc/system/local/ (hopefully it was in $SPLUNK_HOME/etc/system/MyApp/default/). In any case, wherever that was, just copy that subdirectory as-is from a working server to a non-working server. In particular, I suspect you have a mismatch with outputs.conf. Also pay careful attention to server.conf which may include configurations that are supposed to be host/server specific and should*NOT* be copied without being adjusted. It might help to do a search like this from your search head (lets' say your hostname is "myhost" and your IP address is "172.0.0.1"):

index=_internal host=hostname OR host=172.0.0.1 OR hostname OR 172.0.0.1

Also, login to a broken forwarder with your browser and search locally for problems like this:

index=_internal WARN or ERR*

pwhitebe
Engager

Alas, the outputs.conf was in 'SPLUNK_HOME/etc/system/local/' but it was identical to the outputs.conf of the other forwarders.

The thing that was also the same were the inputs.conf and the server.conf files: They were identical to one of the earlier servers.

So it looks like my rush to copy directories (although I might have copied an entire splunk-forwarder installation and not just the local directory) is what is causing all the trouble: They missing 6 were reporting, just under a hostname of another box.

So, I updated the hostnames and things suddenly started appearing under those new hostnames, in addition to the 10 I was seeing before.

Many thanks for pointing me to a new place to look, woodcock.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...