splunkforwarder-8.1.0 New installation on Ubuntu18 error

[batchenr]

Hey,

I have just download and try to install  splunkforwarder but it gives me an error:

#download package like this:

wget -O splunkforwarder-8.1.0-f57c09e87251-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.1.0&product=universalforwarder&filename=splunkforwarder-8.1.0-f57c09e87251-linux-2.6-amd64.deb&wget=true'

dpkg -i splunkforwarder-8.1.0-f57c09e87251-linux-2.6-amd64.deb


(Reading database ... 118207 files and directories currently installed.)
Preparing to unpack splunkforwarder-8.1.0-f57c09e87251-linux-2.6-amd64.deb ...
This looks like an upgrade of an existing Splunk Server. Attempting to stop the installed Splunk Server...
splunkd is not running.
Unpacking splunkforwarder (8.1.0) over (8.1.0) ...
Setting up splunkforwarder (8.1.0) ...
cp: cannot stat '/opt/splunkforwarder/etc/regid.2001-12.com.splunk-UniversalForwarder.swidtag': No such file or directory
complete

machine info :

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

What should i do ?

[burwell]

Could install from the tar file in the interim? 

https://docs.splunk.com/Documentation/Forwarder/8.1.0/Forwarder/Installanixuniversalforwarder#Instal...

Also file a bug with Splunk support.

[Richfez]

For the first thing - the TGZ installation method is literally the first one listed in the installation documents:

https://docs.splunk.com/Documentation/Splunk/8.1.0/Installation/InstallonLinux

For the second thing (networking error)

In any case, your error is cause by a misconfiguration of one of the below:

a) Your local or network firewall

b) Your system's network configuration

c) Your network's configuration

"No route to host" is a standard error meaning that your system has no "route" through which it can send networking packets to your Splunk server.  If it can't send networking packets, it can't send Splunk data.  This has nothing to do with Splunk, as you can prove by opening up a terminal on the UF and typing 'ping X.X.X.X' substituting your splunk server's IP there.  That should give you an error very similar to the one you see in the Splunk logs.

To resolve, open up Google, or Bing, or DuckDuckGo or any other search engine and type in "no route to host".  When you get results, read a few, think about how the things they describe might be the case, and investigate.  It might take looking at a few sites, following a few commands using netstat and traceroute to figure out where the problem is, and revising your searches and approach to fixing it.

To open a ticket, go to the Support Portal for Splunk and click create ticket, or whatever that button is.  Since you are talking about a UF, I assume you have a Splunk server with probably an actual license installed on it, so you should have support.  If that's not the case, well,  I hope a) you aren't using this for production purposes and b) maybe you should pay for support... ?

If you tried to open on on "no route to host" they're probably going to tell you to "fix your networking" 🙂   But the failure to be able to install might get some traction with them.

-Rich

[Richfez]

A short term solution - and the one I  use anyway - is to use the .tgz installer.  It takes the additional steps of a) creating a user, b) chown -R on the /opt/splunk directory to change it's ownership to that new user, and c) setting up boot-enable manually.  But it's all easy enough. 

I do not believe it's recommended to mix and match a .deb install with a .tgz upgrade or vice versa, so you may want to use this with a little bit of caution to make sure you know *this* forwarder has the .tgz on, but otherwise it should work OK.

Also why not open a support ticket for this?  Sounds like an appropriate step.

[batchenr]

What do you mean "the .tgz installer." any links to that or a guide please?

i did the chown -R  with user splunk  and boot-enable

I see in the logs :

11-23-2020 09:27:13.491 +0000 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
11-23-2020 09:27:13.491 +0000 INFO CascadingReplicationManager - Using value for property max_replication_threads=2.
11-23-2020 09:27:13.491 +0000 INFO CascadingReplicationManager - Using value for property max_replication_jobs=5.
11-23-2020 09:27:13.514 +0000 INFO TcpOutputProc - Removing quarantine from idx=X.X.X.X:9997
11-23-2020 09:27:13.514 +0000 WARN TcpOutputFd - Connect to X.X.X.X:9997 failed. No route to host
11-23-2020 09:27:13.514 +0000 ERROR TcpOutputFd - Connection to host=X.X.X.X:9997 failed
11-23-2020 09:27:13.515 +0000 WARN TcpOutputFd - Connect to X.X.X.X:9997 failed. No route to host
11-23-2020 09:27:13.515 +0000 ERROR TcpOutputFd - Connection to host=X.X.X.X:9997 failed
11-23-2020 09:27:13.515 +0000 WARN TcpOutputProc - Applying quarantine to ip=X.X.X.X port=9997 _numberOfFailures=2

I cant make it start and FW is ok. we have checked i think i, installing it wrong and all guides are the same.. i need some guide.

also i don't know how to open a ticket.

==============================

Full seconds iinstall from tgz same results:

tar xvzf splunkforwarder-8.1.0-f57c09e87251-Linux-x86_64.tgz -C /opt

chown -R splunk:splunk /opt/splunkforwarder/

/opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 0 --accept-license --answer-yes --no-prompt --seed-passwd XXX

./splunk start

Splunk> Australian for grep.

Checking prerequisites...

        Checking mgmt port [8089]: open

                Creating: /opt/splunkforwarder/var/lib/splunk

                Creating: /opt/splunkforwarder/var/run/splunk

                Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n

                Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css

                Creating: /opt/splunkforwarder/var/run/splunk/upload

                Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry

                Creating: /opt/splunkforwarder/var/spool/splunk

                Creating: /opt/splunkforwarder/var/spool/dirmoncache

                Creating: /opt/splunkforwarder/var/lib/splunk/authDb

                Creating: /opt/splunkforwarder/var/lib/splunk/hashDb

New certs have been generated in '/opt/splunkforwarder/etc/auth'.

        Checking conf files for problems...

        Done

        Checking default conf files for edits...

        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-8.1.0-f57c09e87251-linux-2.6-x86_64-manifest'

        All installed files intact.

        Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Done

Same error with logs:

11-23-2020 10:24:40.031 +0000 WARN  TcpOutputFd - Connect toX.X.X.X10:9997 failed. No route to host

[kiragsplunk]

1) Check the port listening "netstat -an | grep 9997" 
2) Under Receive data, click Add new. Add the default forwarder listening port (typically TCP port 9997) and click Save

[batchenr]

Problem was splunk server having its own firewall so we needed to add ip there.

thanks