All Apps and Add-ons

Splunk App for Infrastructure EasyInstall failure

Engager

Hello,
I'm trying to get SAI working on my laptop to sort of kick the tires and hopefully install it at work (I don't want to break our production Splunk). I have a Fedora 30 machine with 20G of memory and 1TB of disk.

I ran the easy install on my laptop, so that my laptop will monitor itself. I'm not seeing any entities at all. Here's what I did, as I followed the installation guide for SAI:

I’m going to do the easy install of “Configure Linux/Unix data collection for Splunk App for Infrastructure”
I created an HEC token. I meet the Prerequisites to configure data collection (yum and all that). But under the “HEC token” section, I see my first mention of “collectd”. Hopefully, I’ll learn more about that!
So now go to the SAI user interface.

OK, again following the instructions, it looks like I’m going to run this ginormous one-liner, which I ran and it looked successful, but I never got an entity. Here's what happened:

export SPLUNK_URL=127.0.0.1 && export HEC_PORT=8088 && export RECEIVER_PORT=9997 && export INSTALL_LOCATION=/opt/ && export HEC_TOKEN=HEC-TOKEN-VALUE-ABCDEFGHIJKLKMN && export SAI_ENABLE_DOCKER= && export DIMENSIONS= METRIC_TYPES=cpu,uptime,df,disk,interface,load,memory,processmon METRIC_OPTS=cpu.by_cpu LOG_SOURCES=/etc/collectd/collectd.log%collectd,\$SPLUNK_HOME/var/log/splunk/*.log*%uf,/var/log/syslog%syslog,/var/log/daemon.log%syslog,/var/log/auth.log%syslog AUTHENTICATED_INSTALL=Yes && wget --no-check-certificate http://127.0.0.1:8000/static/app/splunk_app_infrastructure/unix_agent/unix-agent.tgz && tar -xzf unix-agent.tgz || gunzip -c unix-agent.tgz | tar xvf - && cd unix-agent && bash install_uf.sh && bash install_agent.sh && cd .. && rm -rf unix-agent && rm -rf unix-agent.tgz

Splunk is nice and tells me that selinux may rain on my parade. There’s a nice URL provided at http://docs.splunk.com/Documentation/InfraApp/2.0.0/Admin/SELinux . So I to made selinux permissive for collectd…

It did say this during the install, and I’m not sure why. Note that Splunk changed ports for me, I did not enter "y" or anything else:

Checking prerequisites...
Checking mgmt port [8089]: not available
ERROR: mgmt port [8089] - port is already bound. Splunk needs to use this port.
Would you like to change ports? [y/n]: y
Enter a new mgmt port:
Setting mgmt to port: 8090
The server's splunkd port has been changed.
Checking mgmt port [8090]: open
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb

...collectd DID complain about not being able to connect to port 8088, so I edited /etc/connectd.conf and changed the port to 8089 and now it doesn't complain, but I STILL don't see any entities connecting.

Here’s my listeners:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:9997            0.0.0.0:*               LISTEN      6943/splunkd        
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      1140/cupsd          
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      6943/splunkd        
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      20907/splunkd       
tcp        0      0 0.0.0.0:8191            0.0.0.0:*               LISTEN      6990/mongod         
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      6943/splunkd        
tcp        0      0 127.0.0.1:8065          0.0.0.0:*               LISTEN      7126/python3.7      
tcp6       0      0 :::1716                 :::*                    LISTEN      1530/kdeconnectd    
tcp6       0      0 ::1:631                 :::*                    LISTEN 1140/cupsd      

So I’m stuck at https://docs.splunk.com/Documentation/InfraApp/2.0.0/Admin/AddDataLinux

I have restarted both collectd and splunkd.

Splunk Employee
Splunk Employee

Check you Settings->Data Inputs -> HTTP Event Collector -> Global Settings..
WHat are the settings here?

What is HTTP Port Number? Are all tokens enabled?