How to run multiple universal forwarders on a single Linux host?

[tusharsaran1]

I am trying to install 2 universal forwarders on a single Linux host.
I read a few articles and changed the httpport and mgmtHostPort in the web.conf file in $SPLUNK_HOME/etc/system/local.
I also changed serverName in server.conf file.
On trying to start the new splunk instance, it gives below error:

The splunk daemon (splunkd) is already running.            [FAILED]

Is there something else that needs to be changed or is there a different method of running 2 universal forwarders on 1 host?

[kamal_jagga]

Hi Guys,

My both the instances of Splunk forwarder are running but I am not able to set separate "host name" for the instances. I have done both the below mentioned changes but abc01-2 is only appearing in "instance name" field of Deployment Server but no changes happened for the "host name".

Updates made in etc/system/local files of both the instances of the forwarder.
1.Splunk1
Inputs.conf
host = abc01

server.conf
[general]
serverName = abc01

2.Splunk2
Inputs.conf
host = abc01-2

server.conf
[general]
serverName = abc01-2

Kindly advise.

[DavidHourani]

Hi @kamal_jagga,

There is no need to change the hostname if its for DS management, simply change the clientName in deployementclient.conf. That name will then show up on your DS making management a lot easier.

Cheers,
David

[woodcock]

Not possible unless you switch to Windows.

[jwelch_splunk]

I downvoted this post because this is a bad idea.

[woodcock]

You are a silly person and not very nice, but I am not downvoting your comment even though you are VERY wrong.

[mattymo · Accepted solution]

DISCLAIMER: AS FAR AS I KNOW YOU WILL NOT MAKE FRIENDS IN SPLUNK SUPPORT IF YOU DO THIS IN PROD AND HAVE ISSUES. NOT GOING AS FAR AS SAYING IT IS UNSUPPORTED (STILL LOOKING TO CONFIRM), BUT YOUR MILEAGE MAY VARY IF YOU HAVE ISSUES AND LOOK TO SUPPORT TO BAIL YOU OUT. USE PARALLELIZATION IF YOU NEED MORE PIPELINES/THRUPUT. CREATE APPS TO SEGREGATE CONFIG MANAGEMENT

http://docs.splunk.com/Documentation/Splunk/6.5.3/Capacity/Parallelization
http://dev.splunk.com/view/get-started/SP-CAAAESC

I was able to successfully install 2 6.5.3 UFs on a Centos 7 box by doing the following:

Step 1 - As root, cd to /opt and wget UF download.

Step 2 - tar -zxvf splunkforwarder-6.5.3-36937ad027d4-Linux-x86_64.tgz

Step 3 - mv splunkforwarder splunkforwarder2

Step 4 - tar -zxvf splunkforwarder-6.5.3-36937ad027d4-Linux-x86_64.tgz

Step 5 - chown -R splunker:splunker *

Step 6 - cd /opt/splunkforwarder2/etc/system/local and vi server.conf

[general]
serverName = splunkforwarder2

Step 7 - vi web.conf

[settings]
httpport = 9000
mgmtHostPort = 127.0.0.1:8099

Step 8 - cd /opt/splunkforwarder/bin and run ./splunk start --accept-license --answer-yes

Step 9 - cd /opt/splunkforwarder2/bin and run ./splunk start --accept-license --answer-yes

Step 10 - verify with ps -ef | grep splunkd

[splunker@n00b-splkufw-01 local]$ ps -ef | grep splunkd
splunker 9992 1 0 18:16 ? 00:00:00 splunkd -p 8089 start
splunker 10000 9992 0 18:16 ? 00:00:00 [splunkd pid=9992] splunkd -p 8089 start [process-runner]
splunker 10073 1 0 18:17 ? 00:00:00 splunkd -p 8099 start
splunker 10081 10073 0 18:17 ? 00:00:00 [splunkd pid=10073] splunkd -p 8099 start [process-runner]
splunker 10145 9431 0 18:21 pts/0 00:00:00 grep --color=auto splunkd

One slight differences in your configs, I left out the [default] stanza in server.conf, not needed.

Couple things to note, I started the UFs in sequence. and I have confirmed I can run ./btool fine:

cd /opt/splunkforwarder/bin

[splunker@n00b-splkufw-01 bin]$ ./splunk btool server list general --debug
/opt/splunkforwarder/etc/system/local/server.conf [general]
/opt/splunkforwarder/etc/system/default/server.conf access_logging_for_phonehome = true
/opt/splunkforwarder/etc/system/default/server.conf allowRemoteLogin = requireSetPassword
/opt/splunkforwarder/etc/system/default/server.conf connectUsingIpVersion = auto
/opt/splunkforwarder/etc/system/default/server.conf defaultHTTPServerCompressionLevel = 6
/opt/splunkforwarder/etc/system/default/server.conf hangup_after_phonehome = false
/opt/splunkforwarder/etc/system/default/server.conf instanceType = download
/opt/splunkforwarder/etc/system/default/server.conf listenOnIPv6 = no
/opt/splunkforwarder/etc/system/default/server.conf parallelIngestionPipelines = 1
/opt/splunkforwarder/etc/system/local/server.conf pass4SymmKey =
/opt/splunkforwarder/etc/system/local/server.conf serverName = n00b-splkufw-01.n00blab.local
/opt/splunkforwarder/etc/system/default/server.conf sessionTimeout = 1h
/opt/splunkforwarder/etc/system/default/server.conf skipHTTPCompressionAcl = 127.0.0.1 ::1
/opt/splunkforwarder/etc/system/default/server.conf tar_format = gnutar
/opt/splunkforwarder/etc/system/default/server.conf useHTTPClientCompression = false
/opt/splunkforwarder/etc/system/default/server.conf useHTTPServerCompression = true

cd /opt/splunkforwarder2/bin

[splunker@n00b-splkufw-01 bin]$ ./splunk btool server list general --debug
/opt/splunkforwarder2/etc/system/local/server.conf [general]
/opt/splunkforwarder2/etc/system/default/server.conf access_logging_for_phonehome = true
/opt/splunkforwarder2/etc/system/default/server.conf allowRemoteLogin = requireSetPassword
/opt/splunkforwarder2/etc/system/default/server.conf connectUsingIpVersion = auto
/opt/splunkforwarder2/etc/system/default/server.conf defaultHTTPServerCompressionLevel = 6
/opt/splunkforwarder2/etc/system/default/server.conf hangup_after_phonehome = false
/opt/splunkforwarder2/etc/system/default/server.conf instanceType = download
/opt/splunkforwarder2/etc/system/default/server.conf listenOnIPv6 = no
/opt/splunkforwarder2/etc/system/default/server.conf parallelIngestionPipelines = 1
/opt/splunkforwarder2/etc/system/local/server.conf pass4SymmKey =
/opt/splunkforwarder2/etc/system/local/server.conf serverName = splunkforwarder2
/opt/splunkforwarder2/etc/system/default/server.conf sessionTimeout = 1h
/opt/splunkforwarder2/etc/system/default/server.conf skipHTTPCompressionAcl = 127.0.0.1 ::1
/opt/splunkforwarder2/etc/system/default/server.conf tar_format = gnutar
/opt/splunkforwarder2/etc/system/default/server.conf useHTTPClientCompression = false
/opt/splunkforwarder2/etc/system/default/server.conf useHTTPServerCompression = true

Hope that helps! Learning purposes only!!!

[mattymo]

Hey all! The year is 2020 and containers have taken over the world.

If you are looking to run multiple forwarders, use docker or kubernetes!

https://github.com/splunk/docker-splunk

[woodcock]

IMHO, there are very few GOOD reasons to run multiple forwarders so please do explain EXACTLY WHY you think that you need to do this. For example, the AQ/AEQ is single-threaded and so if you have even a modest number of .tar files to forward, you will have to install multiple forwarders on the same server. Tell us *EXACTLY why you are taking this route because I suspect it is unnecessary.

[tusharsaran1]

Here is our use case:
We have 2 separate log locations that we want to scan - Local logs that exist on each host and an NFS mounted log directory (containing 500+ sub directories).
We want to have a local UF running on each host that will scan the local logs on that host.
We'll have a 2nd UF that will run on only 1 host in every data center that will scan the 500+ NFS log directories.
So in every data center, we will have 1 host where we will be running 2 variants of UF. The reason why we cant have a single UF scanning both local and NFS logs is because that will result in multiple indexed copies of the same NFS logs.
Is there a better way of implementing this?

[woodcock]

You do not need 2 forwarders for that. Just create a new serverclass on your DS called something like SharedNFSCaptain and also an app of the same name and then add just one of the forwarder's IP addresses to whitelist. Then you can easily move this function from one host to another just by changing the whitelist AND you are using a single forwarder instance on every host.

[tusharsaran1]

Thanks for the explanation. I have started creating apps/server classes on our deployment server in POC environment. This approach is going to work for 90% of our hosts.
However, for the remaining 10%, we would still need 2 UFs on 1 host because these hosts are already having an instance of UF that is managed and owned by a separate team outside our line of business. If we are using these hosts as deployment clients, is it possible to specify which instance of UF is managed by our DS?

[woodcock]

Each instance can be managed by a separate DS but any instance may be managed only by exactly 1 DS.

[tusharsaran1]

That would work. When adding this host as a client in DS, how do I point the DS to look at a particular UF instance on that host?

[tusharsaran1]

The last question can be ignored. I figured out the answer myself.

[mattymo]

Can that 1 host live on a VM?

[tusharsaran1]

No, it will be a physical machine.

[woodcock]

Make sure that you are using the correct Splunk install, in your case the Universal Forwarder (it appears that you are using the full Splunk Enterprise download). Next, install each one in a DIFFERENT directory. Then make sure that each instance has a different value for serverName inside of server.conf:

[general]
serverName = ThisMustBeUniqueForEachInstance

[jwelch_splunk]

I downvoted this post because i don't really feel this is an answer, but really fits better under comments. if after working with op it proves this is a solution it could be converted.

[tusharsaran1]

I have already changed the serverName. The 2 different installations are in /opt/splunkforwarder and /opt/splunkforwarder2. The ports on the 2 instances are also different. What else could be causing this issue?

[tusharsaran1]

Another observation - I am able to start splunkforwarder2 on port 8099 if I change the port via CLI. However, adding a line in web.conf is not working. Any idea what could be causing this?