For context, I attempted to install the Splunk App for Unix and Linux, and when I uploaded the .tgz file to the indexer, I was greeted with an error which I unfortunately didn't save.
Attempted Troubleshooting:
1. Curling the website before visiting on a browser is successful, though returns an untrusted certificate error (running Splunk over TLS and so this is normal for my deployment).
2. After visiting the web interface on either Chrome or Firefox, in and out of incognito, the browser attempts connection for a while and then says "unable to find web page"
3. Any subsequent curls from the indexer machine result in refused connections.
4. Restarting the splunk daemon resets the entire sequence where curls are successful and the web interface runs until I visit it in a browser.
5. The Splunk deployment was fully operational before I attempted to install the Splunk App for Unix and Linux which rules out any machine, and networking problems.
Environment Information:
1. Splunk Enterprise on a dev license
2. Indexer runs in Centos 8 VM on a ESXI 6.5 host.
Questions related to problem:
1. Has anyone experienced this problem before?
2. What log files can I look into to get a better idea as to what the problem is?
If you are referring to "Splunk App for Unix and Linux"
"https://splunkbase.splunk.com/app/273/"
Would suggest going to your Splunk instance app directory
/opt/splunk/etc/app
and remove the directory "splunk_app_for_nix"
rm -rf splunk_app_for_nix
and restart Splunk services
You can run a real time search via the CLI to troubleshoot the issue
./splunk rtsearch 'index=_internal' sourcetype=splunkd -earliest_time 'rt-30s' -latest_time 'rt+30s'
If the app install had some issue, you can run the same from the CLI
./splunk search 'index=_internal '' -earliest_time 'time of install' -latest_time 'T+5m' -preview true