I'm trying to get SAI working on my laptop to sort of kick the tires and hopefully install it at work (I don't want to break our production Splunk). I have a Fedora 30 machine with 20G of memory and 1TB of disk.
I ran the easy install on my laptop, so that my laptop will monitor itself. I'm not seeing any entities at all. Here's what I did, as I followed the installation guide for SAI:
I’m going to do the easy install of “Configure Linux/Unix data collection for Splunk App for Infrastructure”
I created an HEC token. I meet the Prerequisites to configure data collection (yum and all that). But under the “HEC token” section, I see my first mention of “collectd”. Hopefully, I’ll learn more about that!
So now go to the SAI user interface.
OK, again following the instructions, it looks like I’m going to run this ginormous one-liner, which I ran and it looked successful, but I never got an entity. Here's what happened:
export SPLUNK_URL=127.0.0.1 && export HEC_PORT=8088 && export RECEIVER_PORT=9997 && export INSTALL_LOCATION=/opt/ && export HEC_TOKEN=HEC-TOKEN-VALUE-ABCDEFGHIJKLKMN && export SAI_ENABLE_DOCKER= && export DIMENSIONS= METRIC_TYPES=cpu,uptime,df,disk,interface,load,memory,processmon METRIC_OPTS=cpu.by_cpu LOG_SOURCES=/etc/collectd/collectd.log%collectd,\$SPLUNK_HOME/var/log/splunk/*.log*%uf,/var/log/syslog%syslog,/var/log/daemon.log%syslog,/var/log/auth.log%syslog AUTHENTICATED_INSTALL=Yes && wget --no-check-certificate http://127.0.0.1:8000/static/app/splunk_app_infrastructure/unix_agent/unix-agent.tgz && tar -xzf unix-agent.tgz || gunzip -c unix-agent.tgz | tar xvf - && cd unix-agent && bash install_uf.sh && bash install_agent.sh && cd .. && rm -rf unix-agent && rm -rf unix-agent.tgz
Splunk is nice and tells me that selinux may rain on my parade. There’s a nice URL provided at http://docs.splunk.com/Documentation/InfraApp/2.0.0/Admin/SELinux . So I to made selinux permissive for collectd…
It did say this during the install, and I’m not sure why. Note that Splunk changed ports for me, I did not enter "y" or anything else:
Checking mgmt port : not available
ERROR: mgmt port  - port is already bound. Splunk needs to use this port.
Would you like to change ports? [y/n]: y
Enter a new mgmt port:
Setting mgmt to port: 8090
The server's splunkd port has been changed.
Checking mgmt port : open
...collectd DID complain about not being able to connect to port 8088, so I edited /etc/connectd.conf and changed the port to 8089 and now it doesn't complain, but I STILL don't see any entities connecting.
Here’s my listeners:
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 6943/splunkd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1140/cupsd tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 6943/splunkd tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN 20907/splunkd tcp 0 0 0.0.0.0:8191 0.0.0.0:* LISTEN 6990/mongod tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 6943/splunkd tcp 0 0 127.0.0.1:8065 0.0.0.0:* LISTEN 7126/python3.7 tcp6 0 0 :::1716 :::* LISTEN 1530/kdeconnectd tcp6 0 0 ::1:631 :::* LISTEN 1140/cupsd
I have restarted both collectd and splunkd.