topic Re: How to run multiple universal forwarders on a single Linux host? in Getting Data In

How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Tue, 18 Apr 2017 15:42:33 GMT

I am trying to install 2 universal forwarders on a single Linux host.
I read a few articles and changed the httpport and mgmtHostPort in the web.conf file in $SPLUNK_HOME/etc/system/local.
I also changed serverName in server.conf file.
On trying to start the new splunk instance, it gives below error:

The splunk daemon (splunkd) is already running.            [FAILED]

Is there something else that needs to be changed or is there a different method of running 2 universal forwarders on 1 host?

Re: How to run multiple universal forwarders on a single Linux host?

mattymo — Tue, 18 Apr 2017 16:05:00 GMT

Should be as simple as unzipping the tar in a different directory, and when you start it from that second bin folder it will ask you to change the ports. Seems to me you are trying to start splunk from the instance thats already running.

Do you mind me asking why you would want to do this? There are better options depending on what your trying to do.

Re: How to run multiple universal forwarders on a single Linux host?

woodcock — Tue, 18 Apr 2017 16:50:52 GMT

Make sure that you are using the correct Splunk install, in your case the Universal Forwarder (it appears that you are using the full Splunk Enterprise download). Next, install each one in a DIFFERENT directory. Then make sure that each instance has a different value for serverName inside of server.conf:

[general]
serverName = ThisMustBeUniqueForEachInstance

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 07:44:54 GMT

Like I mentioned, we have already changed the ports and the server name but its still not working. The reason why we want to do this is because we want 2 different flavours of universal forwarder on 1 host. Each one will be scanning a different set of directories.

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 07:46:45 GMT

I have already changed the serverName. The 2 different installations are in /opt/splunkforwarder and /opt/splunkforwarder2. The ports on the 2 instances are also different. What else could be causing this issue?

Re: How to run multiple universal forwarders on a single Linux host?

tlam_splunk — Wed, 19 Apr 2017 09:07:51 GMT

I think you have done the necessary change. You could check the process status (ps) to make sure that there is no previous Splunk running process when you start. Also, after startup the first one, check to see it's using the managment port you set.

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 09:54:10 GMT

I stopped the forwarder that was running already and then tried to start the new one.
I have modified web.conf in /opt/splunkforwarder2/etc/system/local directory with the below details:
[default]
[settings]
httpport = 5000
mgmtHostPort = 127.0.0.1:8099

When I try to start splunkforwarder2, it is still using mgmt port 8089. Is it possible that its still reading the configuration from /opt/splunkforwarder rather than /opt/splunkforwarder2?

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 10:04:14 GMT

Another observation - I am able to start splunkforwarder2 on port 8099 if I change the port via CLI. However, adding a line in web.conf is not working. Any idea what could be causing this?

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 10:29:10 GMT

I have confirmed my findings. Changing mgmt port via CLI in /opt/splunkforwarder2 still makes changes in /opt/splunkforwarder/etc/system/local.web.conf
There must be a way to decouple the 2 directories so that they look at their own config directories.

Re: How to run multiple universal forwarders on a single Linux host?

mattymo — Wed, 19 Apr 2017 10:30:35 GMT

Why do you need different flavours? I still question the validity of this set up if it's anything being used in a prod environment...but anyways....

can you please run ./splunk btool web list --debug so we can confirm your edits are correct and being picked up? You said you were able to successfully configure the second instance by using cli so perhaps you have an error in your config? btool will tell us whether it sees ur changes

be mindful to ensure you are in the correct bin dir when calling the start. I assume u are doing that...

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 10:44:15 GMT

I thought I was able to correctly configure the 2nd instance through CLI but I was wrong. As mentioned in my previous comment, CLI commands are still making changes in the config directory of the original installation.
I ran ./splunk set splunkd-port 8099 in the directory /opt/splunkforwarder2/bin
It changed mgmtPort in /opt/splunkforwarder/etc/system/local/web.conf

How can I avoid this?

Re: How to run multiple universal forwarders on a single Linux host?

mattymo — Wed, 19 Apr 2017 10:45:45 GMT

hmm maybe an environment variable thing?

How did you install? tarball? what command are you running and from which path to change the port?

Re: How to run multiple universal forwarders on a single Linux host?

mattymo — Wed, 19 Apr 2017 10:47:37 GMT

by editting the config files manually I would guess. Its gotta be the environment variables or something...

http://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/Changedefaultvalues

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 10:51:49 GMT

yeah it certainly appears to be an env variable issue.

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 10:54:21 GMT

I tried exporting $SPLUNK_HOME as /opt/splunkforwarder2 but it didnt make a difference. Changes are still getting done in /opt/splunkforwarder.
Manually changing the config files in /opt/splunkforwarder2/etc/system/local isnt helping either (probably because this instance is somehow reading config from /opt/splunkforwarder/etc/system/local).

Regarding installation, I did the first one through rpm and then copied /opt/splunkforwarder to /opt/splunkforwarder2 to create a second directory containing all the binaries and other sub-directories.

Re: How to run multiple universal forwarders on a single Linux host?

mattymo — Wed, 19 Apr 2017 11:08:57 GMT

I'll try in my lab later in the name of science lol but i maintain this is a bad idea and I have not seen any truly valid use cases to support doing this.

a uf is more than capable of monitoring 2 seperate directories and routing said inputs to seperate destinations even!

Is this a classic case of 2 different splunk indexing instances/teams wanting to monitor the same host?

what nix os are u running?

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 11:28:44 GMT

Its not just 2 directories that we are monitoring. There are 2 different set of directories (one of which will be ~500 directories).
Also, we want to manage the configuration separately.
We are running it on RHEL6.7

Re: How to run multiple universal forwarders on a single Linux host?

mattymo — Wed, 19 Apr 2017 13:28:58 GMT

Well, i have done all I could to save you from the pain lol.

Standing by the advice to not go this route, but will try the config on my centos box later.

Re: How to run multiple universal forwarders on a single Linux host?

woodcock — Wed, 19 Apr 2017 14:28:28 GMT

IMHO, there are very few GOOD reasons to run multiple forwarders so please do explain EXACTLY WHY you think that you need to do this. For example, the AQ/AEQ is single-threaded and so if you have even a modest number of .tar files to forward, you will have to install multiple forwarders on the same server. Tell us *EXACTLY why you are taking this route because I suspect it is unnecessary.

Re: How to run multiple universal forwarders on a single Linux host?

tusharsaran1 — Wed, 19 Apr 2017 16:06:56 GMT

Here is our use case:
We have 2 separate log locations that we want to scan - Local logs that exist on each host and an NFS mounted log directory (containing 500+ sub directories).
We want to have a local UF running on each host that will scan the local logs on that host.
We'll have a 2nd UF that will run on only 1 host in every data center that will scan the 500+ NFS log directories.
So in every data center, we will have 1 host where we will be running 2 variants of UF. The reason why we cant have a single UF scanning both local and NFS logs is because that will result in multiple indexed copies of the same NFS logs.
Is there a better way of implementing this?