How to install Splunk Universal Forwarder on a Linux server?

[molinarf]

I know this was probably answered before, but I am not able to find any answers...

I am trying to install the Splunk UF on a Linux server after having to manually uninstall it because of overlapping 7.2.3 (.tgz) and 8.1.0 (.rpm) packages. I am trying to install the 8.1.0 rpm but get the error that it is already installed. When I try to uninstall it since the error says it's installed, then it says that it is already installed. I can't reboot the server because of operations, but would like to have Splunk operational and reporting to the indexer. Can anyone help with guidance on how to overcome this error?

Thank you for any assistance that can be provided.

[gordo32]

I had a similar problem to this in the past, and if I remember correctly I downloaded the tgz file and did:

sudo tar xvzf /tmp/splunk.tgz -C /opt

This overwrote, and I was able to do "splunk disable boot-start", etc...

[molinarf]

Per Support, I did do the installation using the tar file

sudo tar xvzf /tmp/splunk.tgz

Splunk is there, just like when I do the .rpm. The problem I am faced with is that it isn't communicating with the indexer. I may have to just remove the UF all together until that server is rebuilt.

[molinarf]

I had done splunk disable boot-start before I uninstalled. The problem was that there was the .tgz and .rpm both installed at the same time. I am trying to clean it up, so that I can properly reinstall. Thanks for the suggestion.

[scelikok]

Hi @molinarf,

Did you try upgrading with "rpm -U" ? 

[molinarf]

That doesn't work... It thinks that it is already installed and I can't uninstall it either. I am finding that the only option is to install a version that wasn't previously installed and see how that goes. If it fails, then I will have to submit a case.

[isoutamo]

Are you log in as root or use sudo?
What rpm -qi splunk\* shows?
r. Ismo

[molinarf]

After running the command  'rpm -qi splunk\* ', the response was

package splunk is not installed.

[isoutamo]

Can you try also rpm -qa | egrep splunk and if it found splunk<something> (splunkforwarder....) then it is installed otherwise probably not via rpm. Also put could try rpm -qif /opt/splunkforwarder/bin/splunk or where ever it has installed.
And how about the user id which you are using?

[molinarf]

Here is the results:

rpm -qa | egrep splunk

splunkforwarder-8.1.1-08187535c166.x86_64

rpm -qif /opt/splunkforwarder/bin/splunk

Name: splunkforwarder

Version: 8.1.1

Release: 8.1.1-08187535c166

etc.....

If these commands show that it is installed, why then when I try to do an uninstall to clean up, it tells me it is not installed. Of course, that is why when I try to reinstall, I get the it is already installed.

Thanks!

[isoutamo]

Yes this means that this package is already installed.

Currently better way to install/update it is use yum instead of rpm or def on redhat 8+ versions.

As it it already installed you should do:

sudo yum update ./splunkforwarder-8.x.y-xxxx

where x and/or y is greater than 1 and 1.

If you are still using old os and use rpm then the correct way is 

sudo rpm -Uvh --replacepkgs (or --force) ./splunkforwarder-8.x.y-xxxx.rpm

man rpm on that box or google will help you more.

[molinarf]

I ran the following command as you suggested.

sudo rpm -Uvh --replacepkgs (or --force) ./splunkforwarder-8.x.y-xxxx.rpm

It seemed to install fine, but it never reported into the indexer. I did some poking around in man rpm and decided to run

rpm -evv --test splunkforwarder-8.x.y-xxxx.rpm

What returned looked like it would uninstall fine. I checked the splunkd.log to see if there was any indications or error related to the installation or in fact anything and I found none, so my assumption is that everything is supposed to be okay. My next step was to stop splunk and then uninstall. I ran the uninstall command from /opt/splunkforwarder/bin since I was there

rpm -e splunkforwarder-8.x.y-xxxx.rpm

I received the following message: Warning: file /opt/splunkforwarder/ftr: remove failed: no such file or directory. I guess I was in the wrong directory so I reran the command from /opt and this error appeared

Error returned "package splunkforwarder-8.x.y-xxxx.rpm is not installed.  Anyway, I looked in the directory and the splunkforwarder directory was there. I checked it and found only etc and var was there. So I am not sure if it would be prudent for me to remove the splunk forwarder directory and reinstall, like maybe using tar splunkforwarder-8.x.y-xxxx.tgz? I look forward to your response. Thank you

[isoutamo]

When you are removing package, don't use .rpm on name, just package name and if needed version etc. Also don't be on a directory which belongs to package when you are trying to remove it. It usually try to remove that directory too.

https://www.tecmint.com/20-practical-examples-of-rpm-commands-in-linux/

And always use sudo rpm or first switch to user root. 

I hope that these instructions helps you.

And finally don't install with tar until you have removed package with rpm, otherwise you have invalid rpm db catalog on your host and it will generate later other issues!

r. Ismo

[molinarf]

I did try that too. Even when I do the rpm -e splunk8.x.y... it still told me that it was not installed. Support had me do the install with the tar file. I now have the UF installed, but the problem now is that the it is not communicating with the indexer.

[isoutamo]

Have you create outputs.conf or app which define where your indexer(s) is/are?or hav you add DS information to this UF?

[molinarf]

There is an outputs.conf file which is showing that there is an indexer and listening on port 9997, but it is showing as inactive. There is no DS (Deployment server) that is listed.

[gordo32]

I'm not sure if you're now trying to solve the installation issue or the fact that you're not getting data into Splunk. Assuming the latter of the two, and since I don't know your level of experience with splunk, my apologies in advance if you've already done the below.

There are 2 things I check when unexpectedly not getting data:

1. On your indexer:

index=_internal host=<host/ip>

Why: When you have a correctly configured outputs.conf, the  universal forwarder's log files should be in the _internal index. If not there, then it's definitely not communicating properly. Use btool to see running config *on the host where universal forwarder is installed*:

<installpath>/bin/splunk btool outputs list | less

2. If  the indexer is getting UF internal logs, run the host with UF:

<installpath>/bin/splunk btool inputs list | more

Why: Make sure the running configuration for your UF is picking up the desired inputs, and not inheriting a setting from somewhere unexpected. Find the stanza for the input you're trying to get an make sure there isn't something inherited from the add-on's default settings that's interfering (especially "disabled = 0" or "disabled = true"). Typos in the Stanza declaration are also common (i.e. [monitor:///var/log/.../*.log])

BTW, you can add a --debug to the above command to find the source files for each config setting:

<installpath>/bin/splunk btool --debug inputs list | more

Make sure to *never* edit anything in /default/*.conf files. Always create a matching file in /local/*.conf and override the relevant stanza settings there.

[molinarf]

the first issue was that I could not uninstall because there were two versions installed... .tgz version 7.3 and .rpm 8.1.1.

Here are the results for each one that you asked in regards to the communication.

index=_internal host=<host/ip>    There was nothing returned.

 btool outputs list | less    tcpout-server://x.x.x.103:9997

btools inputs list | more

 host=$decideOnStartup$

connection_host=ip

index=default

[gordo32]

You should have 2 stanzas in outputs.conf:

[tcpout]

...bunch of stuff from default.conf...
defaultGroup = MyIndexer

...bunch of stuff from default.conf...

[tcpout:MyIndexer]
server = indexer.domain.com:9997

[molinarf]

here is what is in the outputs.conf file in the /etc/system/local

[tcpout]

defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = x.x.x.103:9997

[tcpout-server://x.x.xx.103:9997]

[gordo32]

That looks fine. Agree with @isoutamo - time to look at UF splunkd.log & metrics.log files.

As an aside, you might want to make sure there aren't any Firewall, HIPS, or other  security tools that might also be blocking outbound connections (and "ps -ef" to make sure splunk is actually running and not just crashing).