topic Re: Number of Forwarders not Increasing? in Deployment Architecture

Number of Forwarders not Increasing?

EllesmereColleg — Thu, 22 Mar 2012 09:36:00 GMT

Hey All,

I have sent out a deployment of the forwarder to a couple hundred machines via GPO. All seemed to be going OK except that since the number of forwarders has reached 108 no further forwarders are appearing. The installs are the same and are completing successfully. Is there a limit on the number of forwarders?

Regards,

Re: Number of Forwarders not Increasing?

ayme — Thu, 22 Mar 2012 09:48:21 GMT

There is no limit from Splunk as such (although there may be limits at the system level based on number of connections to Indexer).

You really want to check the $SPLUNK_HOME/var/log/splunk/splunkd.log on one of the non-communicating Forwarders and see where it complains of not being able to connect.

Re: Number of Forwarders not Increasing?

EllesmereColleg — Thu, 22 Mar 2012 10:21:53 GMT

Ive Checked the outputs.conf on the client and that is pointing to the correct server and port. the splunkd.log says ...

03-22-2012 10:20:14.085 +0000 WARN PubSubConnection - Cannot convert str: to a valid status, returning eRejected.
03-22-2012 10:20:25.804 +0000 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: rejected
03-22-2012 10:20:37.819 +0000 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: rejected
03-22-2012 10:20:49.819 +0000 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: rejected
03-22-2012 10:21:01.819 +0000 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: rejected

Re: Number of Forwarders not Increasing?

EllesmereColleg — Fri, 23 Mar 2012 09:20:12 GMT

Anyone able to suggest why the server would rejecting additional forwarders?

Re: Number of Forwarders not Increasing?

Drainy — Fri, 23 Mar 2012 09:34:28 GMT

Have a read of, http://splunk-base.splunk.com/answers/4097/is-there-a-maximum-number-of-forwarders-per-indexer

Hopefully that should answer your questions 🙂

Re: Number of Forwarders not Increasing?

ayme — Fri, 23 Mar 2012 09:38:22 GMT

The above messages relate to the Deployment Server so you don't want to look at the messages for the "DeploymentClient" (for the time being). Restart the Forwarder and look at the log file again - you should see messages related to "TcpOut".

Re: Number of Forwarders not Increasing?

EllesmereColleg — Fri, 23 Mar 2012 09:52:07 GMT

Unfortunately not, i have 108 clients forwarding successfully to the server, just application and system logs, which is about 4mb a day a day at the moment. Should be plenty of room for more clients from them numbers. Just not sure why the server is rejecting the additional ones.

Re: Number of Forwarders not Increasing?

Drainy — Fri, 23 Mar 2012 09:53:49 GMT

Well, like Ayme says above you need to have a look at the deployment server end to see what it is saying but if the connections are being rejected the chances are its probably a TCP socket issue

Re: Number of Forwarders not Increasing?

EllesmereColleg — Fri, 23 Mar 2012 09:55:14 GMT

16.103 is the correct server and 9997 is the correct port..

03-23-2012 09:53:06.363 +0000 INFO TcpOutputProc - Connected to idx=192.168.16.103:9997
03-23-2012 09:53:11.285 +0000 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: rejected

Re: Number of Forwarders not Increasing?

ayme — Fri, 23 Mar 2012 10:00:51 GMT

Well, certainly looks to be connected. Chances are it just hasn't sent any data. The search:

index=_internal | stats count by host

...should show you that it is reporting internal messages, in which case you just need to ensure you have well configured monitors in the inputs.conf of the "missing" forwarders (could the fact that it is not connecting to the Deployment Server be the reason for this?)

Re: Number of Forwarders not Increasing?

EllesmereColleg — Fri, 23 Mar 2012 10:15:30 GMT

Thanks, the search index=_internal | stats count by host
Does indeed display the missing hosts and the errors. For example

03-23-2012 10:11:43.501 +0000 WARN PubSubConnection - Cannot convert str: to a valid status, returning eRejected.
host=6F01-01-1560 Options| sourcetype=splunkd

Options| source=C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log

Re: Number of Forwarders not Increasing?

EllesmereColleg — Fri, 23 Mar 2012 10:59:39 GMT

I have compared the contents of input.conf in the following location
c:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf

However they both seem the same with the right host. Unless i am looking at the wrong inputs.conf?

A working computer.

Contains
[default]
host = IT1-05-0847

A missing machine
[default]
host = Admiss-02-0674

Re: Number of Forwarders not Increasing?

EllesmereColleg — Fri, 23 Mar 2012 11:47:07 GMT

Strangely, even the ones i can see in the deployment manager as active report that the server refused the connection?