I am trying to install 2 universal forwarders on a single Linux host.
I read a few articles and changed the httpport and mgmtHostPort in the web.conf file in $SPLUNK_HOME/etc/system/local.
I also changed serverName in server.conf file.
On trying to start the new splunk instance, it gives below error:
The splunk daemon (splunkd) is already running. [FAILED]
Is there something else that needs to be changed or is there a different method of running 2 universal forwarders on 1 host?
http://docs.splunk.com/Documentation/Splunk/6.5.3/Capacity/Parallelization
http://dev.splunk.com/view/get-started/SP-CAAAESC
I was able to successfully install 2 6.5.3 UFs on a Centos 7 box by doing the following:
Step 1 - As root, cd to /opt and wget UF download.
Step 2 - tar -zxvf splunkforwarder-6.5.3-36937ad027d4-Linux-x86_64.tgz
Step 3 - mv splunkforwarder splunkforwarder2
Step 4 - tar -zxvf splunkforwarder-6.5.3-36937ad027d4-Linux-x86_64.tgz
Step 5 - chown -R splunker:splunker *
Step 6 - cd /opt/splunkforwarder2/etc/system/local
and vi server.conf
[general]
serverName = splunkforwarder2
Step 7 - vi web.conf
[settings]
httpport = 9000
mgmtHostPort = 127.0.0.1:8099
Step 8 - cd /opt/splunkforwarder/bin
and run ./splunk start --accept-license --answer-yes
Step 9 - cd /opt/splunkforwarder2/bin
and run ./splunk start --accept-license --answer-yes
Step 10 - verify with ps -ef | grep splunkd
[splunker@n00b-splkufw-01 local]$ ps -ef | grep splunkd
splunker 9992 1 0 18:16 ? 00:00:00 splunkd -p 8089 start
splunker 10000 9992 0 18:16 ? 00:00:00 [splunkd pid=9992] splunkd -p 8089 start [process-runner]
splunker 10073 1 0 18:17 ? 00:00:00 splunkd -p 8099 start
splunker 10081 10073 0 18:17 ? 00:00:00 [splunkd pid=10073] splunkd -p 8099 start [process-runner]
splunker 10145 9431 0 18:21 pts/0 00:00:00 grep --color=auto splunkd
One slight differences in your configs, I left out the [default]
stanza in server.conf
, not needed.
Couple things to note, I started the UFs in sequence. and I have confirmed I can run ./btool fine:
cd /opt/splunkforwarder/bin
[splunker@n00b-splkufw-01 bin]$ ./splunk btool server list general --debug
/opt/splunkforwarder/etc/system/local/server.conf [general]
/opt/splunkforwarder/etc/system/default/server.conf access_logging_for_phonehome = true
/opt/splunkforwarder/etc/system/default/server.conf allowRemoteLogin = requireSetPassword
/opt/splunkforwarder/etc/system/default/server.conf connectUsingIpVersion = auto
/opt/splunkforwarder/etc/system/default/server.conf defaultHTTPServerCompressionLevel = 6
/opt/splunkforwarder/etc/system/default/server.conf hangup_after_phonehome = false
/opt/splunkforwarder/etc/system/default/server.conf instanceType = download
/opt/splunkforwarder/etc/system/default/server.conf listenOnIPv6 = no
/opt/splunkforwarder/etc/system/default/server.conf parallelIngestionPipelines = 1
/opt/splunkforwarder/etc/system/local/server.conf pass4SymmKey =
/opt/splunkforwarder/etc/system/local/server.conf serverName = n00b-splkufw-01.n00blab.local
/opt/splunkforwarder/etc/system/default/server.conf sessionTimeout = 1h
/opt/splunkforwarder/etc/system/default/server.conf skipHTTPCompressionAcl = 127.0.0.1 ::1
/opt/splunkforwarder/etc/system/default/server.conf tar_format = gnutar
/opt/splunkforwarder/etc/system/default/server.conf useHTTPClientCompression = false
/opt/splunkforwarder/etc/system/default/server.conf useHTTPServerCompression = true
cd /opt/splunkforwarder2/bin
[splunker@n00b-splkufw-01 bin]$ ./splunk btool server list general --debug
/opt/splunkforwarder2/etc/system/local/server.conf [general]
/opt/splunkforwarder2/etc/system/default/server.conf access_logging_for_phonehome = true
/opt/splunkforwarder2/etc/system/default/server.conf allowRemoteLogin = requireSetPassword
/opt/splunkforwarder2/etc/system/default/server.conf connectUsingIpVersion = auto
/opt/splunkforwarder2/etc/system/default/server.conf defaultHTTPServerCompressionLevel = 6
/opt/splunkforwarder2/etc/system/default/server.conf hangup_after_phonehome = false
/opt/splunkforwarder2/etc/system/default/server.conf instanceType = download
/opt/splunkforwarder2/etc/system/default/server.conf listenOnIPv6 = no
/opt/splunkforwarder2/etc/system/default/server.conf parallelIngestionPipelines = 1
/opt/splunkforwarder2/etc/system/local/server.conf pass4SymmKey =
/opt/splunkforwarder2/etc/system/local/server.conf serverName = splunkforwarder2
/opt/splunkforwarder2/etc/system/default/server.conf sessionTimeout = 1h
/opt/splunkforwarder2/etc/system/default/server.conf skipHTTPCompressionAcl = 127.0.0.1 ::1
/opt/splunkforwarder2/etc/system/default/server.conf tar_format = gnutar
/opt/splunkforwarder2/etc/system/default/server.conf useHTTPClientCompression = false
/opt/splunkforwarder2/etc/system/default/server.conf useHTTPServerCompression = true
Hope that helps! Learning purposes only!!!
Should be as simple as unzipping the tar in a different directory, and when you start it from that second bin folder it will ask you to change the ports. Seems to me you are trying to start splunk from the instance thats already running.
Do you mind me asking why you would want to do this? There are better options depending on what your trying to do.
Like I mentioned, we have already changed the ports and the server name but its still not working. The reason why we want to do this is because we want 2 different flavours of universal forwarder on 1 host. Each one will be scanning a different set of directories.
Why do you need different flavours? I still question the validity of this set up if it's anything being used in a prod environment...but anyways....
can you please run ./splunk btool web list --debug
so we can confirm your edits are correct and being picked up? You said you were able to successfully configure the second instance by using cli so perhaps you have an error in your config? btool will tell us whether it sees ur changes
be mindful to ensure you are in the correct bin dir when calling the start. I assume u are doing that...
Its not just 2 directories that we are monitoring. There are 2 different set of directories (one of which will be ~500 directories).
Also, we want to manage the configuration separately.
We are running it on RHEL6.7
As described you still don't need two UF's. Separate management of config is better achieved by breaking the two inputs or sets of inputs, into 2 apps and then managing those apps separately, possibly via Deployment Server. Likewise one UF can have as many separate inputs as you need, each monitoring different sets of directories. I hope that helps.
Well, i have done all I could to save you from the pain lol.
Standing by the advice to not go this route, but will try the config on my centos box later.
I thought I was able to correctly configure the 2nd instance through CLI but I was wrong. As mentioned in my previous comment, CLI commands are still making changes in the config directory of the original installation.
I ran ./splunk set splunkd-port 8099 in the directory /opt/splunkforwarder2/bin
It changed mgmtPort in /opt/splunkforwarder/etc/system/local/web.conf
How can I avoid this?
by editting the config files manually I would guess. Its gotta be the environment variables or something...
http://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/Changedefaultvalues
yeah it certainly appears to be an env variable issue.
I'll try in my lab later in the name of science lol but i maintain this is a bad idea and I have not seen any truly valid use cases to support doing this.
a uf is more than capable of monitoring 2 seperate directories and routing said inputs to seperate destinations even!
Is this a classic case of 2 different splunk indexing instances/teams wanting to monitor the same host?
what nix os are u running?
I think you have done the necessary change. You could check the process status (ps) to make sure that there is no previous Splunk running process when you start. Also, after startup the first one, check to see it's using the managment port you set.
I stopped the forwarder that was running already and then tried to start the new one.
I have modified web.conf in /opt/splunkforwarder2/etc/system/local directory with the below details:
[default]
[settings]
httpport = 5000
mgmtHostPort = 127.0.0.1:8099
When I try to start splunkforwarder2, it is still using mgmt port 8089. Is it possible that its still reading the configuration from /opt/splunkforwarder rather than /opt/splunkforwarder2?
I have confirmed my findings. Changing mgmt port via CLI in /opt/splunkforwarder2 still makes changes in /opt/splunkforwarder/etc/system/local.web.conf
There must be a way to decouple the 2 directories so that they look at their own config directories.
hmm maybe an environment variable thing?
How did you install? tarball? what command are you running and from which path to change the port?
I tried exporting $SPLUNK_HOME as /opt/splunkforwarder2 but it didnt make a difference. Changes are still getting done in /opt/splunkforwarder.
Manually changing the config files in /opt/splunkforwarder2/etc/system/local isnt helping either (probably because this instance is somehow reading config from /opt/splunkforwarder/etc/system/local).
Regarding installation, I did the first one through rpm and then copied /opt/splunkforwarder to /opt/splunkforwarder2 to create a second directory containing all the binaries and other sub-directories.
I think this is where your issue lies. I did my answer below with tar both times. I bet if you do rpm then tar instead of copy it would have worked
Just to close this off, the RPM issue could have been resolved on the second instance by fixing /opt/splunkforwarder2/etc/launch-conf.conf as the $SPLUNK_HOME and proc names can be accessed there.
Thanks to @jwelch for the reminder.
[splunker@n00b-splkufw-01 etc]$ cat splunk-launch.conf
# Version 6.5.3
# Modify the following line to suit the location of your Splunk install.
# If unset, Splunk will use the parent of the directory containing the splunk
# CLI executable.
#
# SPLUNK_HOME=/home/build/build-home/ivory
# By default, Splunk stores its indexes under SPLUNK_HOME in the
# var/lib/splunk subdirectory. This can be overridden
# here:
#
# SPLUNK_DB=/home/build/build-home/ivory/var/lib/splunk
# Splunkd daemon name
SPLUNK_SERVER_NAME=SplunkForwarder
# Splunkweb daemon name
SPLUNK_WEB_NAME=splunkweb
# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)
#
# SPLUNK_OS_USER[splunker@n00b-splkufw-01 etc]$
AHA! We may have a winner. I never use RPMs and always use the tarballs and it always worked fine for me.
Does Splunk officially support for installing 2 UFs for Linux Machines?? I know it doesn't for windows.