multiple deployment client on same host pointing to different deployment server

[shailesh030]

Hi!,
1. We have a server say Source1 in which two applications write log files into two different directories say /home/abc/loga.txt and logb.txt
2. There is only one instance of splunk forwarder running on this log server.

My goal is to forward these two log entries to two corresponding different splunk indexers (say indexer.abc.com:9000 and indexer.abc.com:9001) running on different VM's. i.e. loga.txt to indexer.abc.com:9000 and logb.txt to indexer.abc.com:9001

To ensure flexibility, best practice and separation of concerns I am trying to achieve this through deploymentservers running on two different instances of searcheads running on different ports. i.e. DS.abc.com:10001 and DS.abc.com:10002.

This is what I tried
1. Created two different client apps under splunkforwarder/etc/apps and configured targeturi of each of them to point to respective deploymentserver
a) splunkforwarder/etc/apps/base_dc1/deploymentclient.conf points to DS.abc.com:10001
b) splunkforwarder/etc/apps/base_dc2/deploymentclient.conf points to DS.abc.com:10002

1. Created deployment-apps in respective deploymentservers DS1 and DS2 so that they can push different inputs.conf and outputs.conf to different apps in Source1 splunk forwarder
   DS1:
   2.1 splunk/etc/deployment-apps/app1/input.conf will forward /home/abc/loga.txt
   2.2 splunk/etc/deployment-apps/app2/output.conf will forward it to indexer.abc.com:9000

   DS2:
   2.1 splunk/etc/deployment-apps/app111/input.conf will forward /home/abc/logb.txt
   2.2 splunk/etc/deployment-apps/app222/output.conf will forward it to indexer.abc.com:9001

2. Created serverclass.conf in DS1 and DS2 and added respective apps

3. Deployed serverclass on DS1 and DS2 respectively

4. Restarted splunkforwarder

I was expecting that on source1 under splunkforwarder, I will see app1,app2,app111 and app222 downloaded but only app1,app2 from 1st deploymentclient.conf got downloaded while app111, app222 did not.

My assumption is that once the 4 apps were downloaded
Using app1/input.conf, app2/output.conf splunkforwarder will forward /home/abc/loga.txt to indexer.abc.com:9000
Using app111/input.conf and app222/output.conf will forward /home/abc/logb.txt to indexer.abc.com:9001

My questions are:

1. Is it possible to have two different deploymentclient.conf under separate apps folder in one splunkforwarder instance running on a server with each of the deploymentclient pointing to a different deploymentserver?If so, what might be wrong with my above steps? splunkd.log on splunkforwarder shows phoneHome connection and apps downloaded for 1st client only.
2. If above is not possible, how can I configure splunkforwarder to forward the two log files to different indexers. I have to make sure any splunk configuration changes for one application log file does not impact the availability of the other.

My constraints:
1. I cannot change port numbers of any of the splunk instances as they are provisioned by a different team.
2. I would prefer to have separate inputs.conf and outputs.conf file for each logs because although logs are on same server, the ownership is across different teams. I cannot afford to involve another application say the one which owns logb while changing parameters related to say loga.txt
3. I cannot have one single deploymentserver with one serverclass.conf & different clientnames due to some organization constraints. Different applications need to maintain their configuration files through their own deploymentserver.

Any help will be highly appreciated

Regards

[lguinn2 · Accepted solution]

A forwarder can only use a single deployment server. Using the configuration precedence rules, one of the configurations will be chosen and any others will be ignored. See Configuration File Precedence for more information.

Perhaps you need to create a single master repository of apps. You could have each organization build and test their own apps and then some process for collecting the apps into the repository. Then you would have a single deployment server for all the forwarders. I don't think that multiple deployment servers is a good option, unless you are able to assign each forwarder to a single deployment server.

[shailesh030]

Thanks lguinn,

Also if I one team decides not to use deployment server but just create apps with config files on client side manually but if another team is using deployment server then whenever 2nd team redeploy's the serverclass the 1st team's apps will be deleted from /splunkforwarder/etc/. Is that a correct statement as well?

This makes life tougher for us. It seems like if there is a server running shared business applications writing logs monitored by single forwarder then there needs to be a central team which owns the deployment server and manages deployment. It takes away deployment independence from individual teams.

[lguinn2]

Each team will need to create apps with unique names. You probably don't want to have multiple teams working on the same app. This will avoid most overwrite/deletion problems.