I am trying to install 2 universal forwarders on a single Linux host.
I read a few articles and changed the httpport and mgmtHostPort in the web.conf file in $SPLUNK_HOME/etc/system/local.
I also changed serverName in server.conf file.
On trying to start the new splunk instance, it gives below error:
The splunk daemon (splunkd) is already running. [FAILED]
Is there something else that needs to be changed or is there a different method of running 2 universal forwarders on 1 host?
Should be as simple as unzipping the tar in a different directory, and when you start it from that second bin folder it will ask you to change the ports. Seems to me you are trying to start splunk from the instance thats already running.
Do you mind me asking why you would want to do this? There are better options depending on what your trying to do.
Like I mentioned, we have already changed the ports and the server name but its still not working. The reason why we want to do this is because we want 2 different flavours of universal forwarder on 1 host. Each one will be scanning a different set of directories.
I think you have done the necessary change. You could check the process status (ps) to make sure that there is no previous Splunk running process when you start. Also, after startup the first one, check to see it's using the managment port you set.
I stopped the forwarder that was running already and then tried to start the new one.
I have modified web.conf in /opt/splunkforwarder2/etc/system/local directory with the below details:
httpport = 5000
mgmtHostPort = 127.0.0.1:8099
When I try to start splunkforwarder2, it is still using mgmt port 8089. Is it possible that its still reading the configuration from /opt/splunkforwarder rather than /opt/splunkforwarder2?
I have confirmed my findings. Changing mgmt port via CLI in /opt/splunkforwarder2 still makes changes in /opt/splunkforwarder/etc/system/local.web.conf
There must be a way to decouple the 2 directories so that they look at their own config directories.
hmm maybe an environment variable thing?
How did you install? tarball? what command are you running and from which path to change the port?
I tried exporting $SPLUNK_HOME as /opt/splunkforwarder2 but it didnt make a difference. Changes are still getting done in /opt/splunkforwarder.
Manually changing the config files in /opt/splunkforwarder2/etc/system/local isnt helping either (probably because this instance is somehow reading config from /opt/splunkforwarder/etc/system/local).
Regarding installation, I did the first one through rpm and then copied /opt/splunkforwarder to /opt/splunkforwarder2 to create a second directory containing all the binaries and other sub-directories.
I think this is where your issue lies. I did my answer below with tar both times. I bet if you do rpm then tar instead of copy it would have worked
AHA! We may have a winner. I never use RPMs and always use the tarballs and it always worked fine for me.