Getting Data In

How to run multiple universal forwarders on a single Linux host?

tusharsaran1
Path Finder

I am trying to install 2 universal forwarders on a single Linux host.
I read a few articles and changed the httpport and mgmtHostPort in the web.conf file in $SPLUNK_HOME/etc/system/local.
I also changed serverName in server.conf file.
On trying to start the new splunk instance, it gives below error:

The splunk daemon (splunkd) is already running.            [FAILED]

Is there something else that needs to be changed or is there a different method of running 2 universal forwarders on 1 host?

1 Solution

mattymo
Splunk Employee
Splunk Employee

DISCLAIMER: AS FAR AS I KNOW YOU WILL NOT MAKE FRIENDS IN SPLUNK SUPPORT IF YOU DO THIS IN PROD AND HAVE ISSUES. NOT GOING AS FAR AS SAYING IT IS UNSUPPORTED (STILL LOOKING TO CONFIRM), BUT YOUR MILEAGE MAY VARY IF YOU HAVE ISSUES AND LOOK TO SUPPORT TO BAIL YOU OUT. USE PARALLELIZATION IF YOU NEED MORE PIPELINES/THRUPUT. CREATE APPS TO SEGREGATE CONFIG MANAGEMENT

http://docs.splunk.com/Documentation/Splunk/6.5.3/Capacity/Parallelization
http://dev.splunk.com/view/get-started/SP-CAAAESC

I was able to successfully install 2 6.5.3 UFs on a Centos 7 box by doing the following:

Step 1 - As root, cd to /opt and wget UF download.

Step 2 - tar -zxvf splunkforwarder-6.5.3-36937ad027d4-Linux-x86_64.tgz

Step 3 - mv splunkforwarder splunkforwarder2

Step 4 - tar -zxvf splunkforwarder-6.5.3-36937ad027d4-Linux-x86_64.tgz

Step 5 - chown -R splunker:splunker *

Step 6 - cd /opt/splunkforwarder2/etc/system/local and vi server.conf

[general]
serverName = splunkforwarder2

Step 7 - vi web.conf

[settings]
httpport = 9000
mgmtHostPort = 127.0.0.1:8099

Step 8 - cd /opt/splunkforwarder/bin and run ./splunk start --accept-license --answer-yes

Step 9 - cd /opt/splunkforwarder2/bin and run ./splunk start --accept-license --answer-yes

Step 10 - verify with ps -ef | grep splunkd

[splunker@n00b-splkufw-01 local]$ ps -ef | grep splunkd
splunker 9992 1 0 18:16 ? 00:00:00 splunkd -p 8089 start
splunker 10000 9992 0 18:16 ? 00:00:00 [splunkd pid=9992] splunkd -p 8089 start [process-runner]
splunker 10073 1 0 18:17 ? 00:00:00 splunkd -p 8099 start
splunker 10081 10073 0 18:17 ? 00:00:00 [splunkd pid=10073] splunkd -p 8099 start [process-runner]
splunker 10145 9431 0 18:21 pts/0 00:00:00 grep --color=auto splunkd

One slight differences in your configs, I left out the [default] stanza in server.conf, not needed.

Couple things to note, I started the UFs in sequence. and I have confirmed I can run ./btool fine:

cd /opt/splunkforwarder/bin

[splunker@n00b-splkufw-01 bin]$ ./splunk btool server list general --debug
/opt/splunkforwarder/etc/system/local/server.conf [general]
/opt/splunkforwarder/etc/system/default/server.conf access_logging_for_phonehome = true
/opt/splunkforwarder/etc/system/default/server.conf allowRemoteLogin = requireSetPassword
/opt/splunkforwarder/etc/system/default/server.conf connectUsingIpVersion = auto
/opt/splunkforwarder/etc/system/default/server.conf defaultHTTPServerCompressionLevel = 6
/opt/splunkforwarder/etc/system/default/server.conf hangup_after_phonehome = false
/opt/splunkforwarder/etc/system/default/server.conf instanceType = download
/opt/splunkforwarder/etc/system/default/server.conf listenOnIPv6 = no
/opt/splunkforwarder/etc/system/default/server.conf parallelIngestionPipelines = 1
/opt/splunkforwarder/etc/system/local/server.conf pass4SymmKey =
/opt/splunkforwarder/etc/system/local/server.conf serverName = n00b-splkufw-01.n00blab.local
/opt/splunkforwarder/etc/system/default/server.conf sessionTimeout = 1h
/opt/splunkforwarder/etc/system/default/server.conf skipHTTPCompressionAcl = 127.0.0.1 ::1
/opt/splunkforwarder/etc/system/default/server.conf tar_format = gnutar
/opt/splunkforwarder/etc/system/default/server.conf useHTTPClientCompression = false
/opt/splunkforwarder/etc/system/default/server.conf useHTTPServerCompression = true

cd /opt/splunkforwarder2/bin

[splunker@n00b-splkufw-01 bin]$ ./splunk btool server list general --debug
/opt/splunkforwarder2/etc/system/local/server.conf [general]
/opt/splunkforwarder2/etc/system/default/server.conf access_logging_for_phonehome = true
/opt/splunkforwarder2/etc/system/default/server.conf allowRemoteLogin = requireSetPassword
/opt/splunkforwarder2/etc/system/default/server.conf connectUsingIpVersion = auto
/opt/splunkforwarder2/etc/system/default/server.conf defaultHTTPServerCompressionLevel = 6
/opt/splunkforwarder2/etc/system/default/server.conf hangup_after_phonehome = false
/opt/splunkforwarder2/etc/system/default/server.conf instanceType = download
/opt/splunkforwarder2/etc/system/default/server.conf listenOnIPv6 = no
/opt/splunkforwarder2/etc/system/default/server.conf parallelIngestionPipelines = 1
/opt/splunkforwarder2/etc/system/local/server.conf pass4SymmKey =
/opt/splunkforwarder2/etc/system/local/server.conf serverName = splunkforwarder2
/opt/splunkforwarder2/etc/system/default/server.conf sessionTimeout = 1h
/opt/splunkforwarder2/etc/system/default/server.conf skipHTTPCompressionAcl = 127.0.0.1 ::1
/opt/splunkforwarder2/etc/system/default/server.conf tar_format = gnutar
/opt/splunkforwarder2/etc/system/default/server.conf useHTTPClientCompression = false
/opt/splunkforwarder2/etc/system/default/server.conf useHTTPServerCompression = true

Hope that helps! Learning purposes only!!!

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

Should be as simple as unzipping the tar in a different directory, and when you start it from that second bin folder it will ask you to change the ports. Seems to me you are trying to start splunk from the instance thats already running.

Do you mind me asking why you would want to do this? There are better options depending on what your trying to do.

- MattyMo
0 Karma

tusharsaran1
Path Finder

Like I mentioned, we have already changed the ports and the server name but its still not working. The reason why we want to do this is because we want 2 different flavours of universal forwarder on 1 host. Each one will be scanning a different set of directories.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Why do you need different flavours? I still question the validity of this set up if it's anything being used in a prod environment...but anyways....

can you please run ./splunk btool web list --debug so we can confirm your edits are correct and being picked up? You said you were able to successfully configure the second instance by using cli so perhaps you have an error in your config? btool will tell us whether it sees ur changes

be mindful to ensure you are in the correct bin dir when calling the start. I assume u are doing that...

- MattyMo
0 Karma

tusharsaran1
Path Finder

Its not just 2 directories that we are monitoring. There are 2 different set of directories (one of which will be ~500 directories).
Also, we want to manage the configuration separately.
We are running it on RHEL6.7

0 Karma

sideview
SplunkTrust
SplunkTrust

As described you still don't need two UF's. Separate management of config is better achieved by breaking the two inputs or sets of inputs, into 2 apps and then managing those apps separately, possibly via Deployment Server. Likewise one UF can have as many separate inputs as you need, each monitoring different sets of directories. I hope that helps.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Well, i have done all I could to save you from the pain lol.

Standing by the advice to not go this route, but will try the config on my centos box later.

- MattyMo
0 Karma

tusharsaran1
Path Finder

I thought I was able to correctly configure the 2nd instance through CLI but I was wrong. As mentioned in my previous comment, CLI commands are still making changes in the config directory of the original installation.
I ran ./splunk set splunkd-port 8099 in the directory /opt/splunkforwarder2/bin
It changed mgmtPort in /opt/splunkforwarder/etc/system/local/web.conf

How can I avoid this?

0 Karma

mattymo
Splunk Employee
Splunk Employee

by editting the config files manually I would guess. Its gotta be the environment variables or something...

http://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/Changedefaultvalues

- MattyMo
0 Karma

tusharsaran1
Path Finder

yeah it certainly appears to be an env variable issue.

0 Karma

mattymo
Splunk Employee
Splunk Employee

I'll try in my lab later in the name of science lol but i maintain this is a bad idea and I have not seen any truly valid use cases to support doing this.

a uf is more than capable of monitoring 2 seperate directories and routing said inputs to seperate destinations even!

Is this a classic case of 2 different splunk indexing instances/teams wanting to monitor the same host?

what nix os are u running?

- MattyMo
0 Karma

tlam_splunk
Splunk Employee
Splunk Employee

I think you have done the necessary change. You could check the process status (ps) to make sure that there is no previous Splunk running process when you start. Also, after startup the first one, check to see it's using the managment port you set.

0 Karma

tusharsaran1
Path Finder

I stopped the forwarder that was running already and then tried to start the new one.
I have modified web.conf in /opt/splunkforwarder2/etc/system/local directory with the below details:
[default]
[settings]
httpport = 5000
mgmtHostPort = 127.0.0.1:8099

When I try to start splunkforwarder2, it is still using mgmt port 8089. Is it possible that its still reading the configuration from /opt/splunkforwarder rather than /opt/splunkforwarder2?

0 Karma

tusharsaran1
Path Finder

I have confirmed my findings. Changing mgmt port via CLI in /opt/splunkforwarder2 still makes changes in /opt/splunkforwarder/etc/system/local.web.conf
There must be a way to decouple the 2 directories so that they look at their own config directories.

0 Karma

mattymo
Splunk Employee
Splunk Employee

hmm maybe an environment variable thing?

How did you install? tarball? what command are you running and from which path to change the port?

- MattyMo
0 Karma

tusharsaran1
Path Finder

I tried exporting $SPLUNK_HOME as /opt/splunkforwarder2 but it didnt make a difference. Changes are still getting done in /opt/splunkforwarder.
Manually changing the config files in /opt/splunkforwarder2/etc/system/local isnt helping either (probably because this instance is somehow reading config from /opt/splunkforwarder/etc/system/local).

Regarding installation, I did the first one through rpm and then copied /opt/splunkforwarder to /opt/splunkforwarder2 to create a second directory containing all the binaries and other sub-directories.

0 Karma

mattymo
Splunk Employee
Splunk Employee

I think this is where your issue lies. I did my answer below with tar both times. I bet if you do rpm then tar instead of copy it would have worked

- MattyMo

mattymo
Splunk Employee
Splunk Employee

Just to close this off, the RPM issue could have been resolved on the second instance by fixing /opt/splunkforwarder2/etc/launch-conf.conf as the $SPLUNK_HOME and proc names can be accessed there.

Thanks to @jwelch for the reminder.

[splunker@n00b-splkufw-01 etc]$ cat splunk-launch.conf
#   Version 6.5.3

# Modify the following line to suit the location of your Splunk install.
# If unset, Splunk will use the parent of the directory containing the splunk
# CLI executable.
#
# SPLUNK_HOME=/home/build/build-home/ivory

# By default, Splunk stores its indexes under SPLUNK_HOME in the
# var/lib/splunk subdirectory.  This can be overridden
# here:
#
# SPLUNK_DB=/home/build/build-home/ivory/var/lib/splunk
# Splunkd daemon name
SPLUNK_SERVER_NAME=SplunkForwarder

# Splunkweb daemon name
SPLUNK_WEB_NAME=splunkweb

# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)
#
# SPLUNK_OS_USER[splunker@n00b-splkufw-01 etc]$ 
- MattyMo
0 Karma

woodcock
Esteemed Legend

AHA! We may have a winner. I never use RPMs and always use the tarballs and it always worked fine for me.

0 Karma

hectorvp
Communicator

Does Splunk officially support for installing 2 UFs for Linux Machines?? I know it doesn't for windows.

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...