Ok, so first lets get some terminology updated here to align with the industrial standard. There is no SF. There is a Splunk Universal Forwarder (UF) and a Splunk Heavy Forwarder (HF). The difference here is that a UF is a minimal agent without a GUI, and a HF is a full instance of Splunk that can act as a forwarder, or it can act as any role in the Splunk environment { Search Head (SH), Deployer, Deployment Server (DS), Indexer (IDX), Monitoring Console (MC), License Master (LM) etc.} Circling back. Can you confirm there are not multiple versions of the UF or HF installed? The behavior you're describing seems very similar to what happens when you have multiple instances installed and in the executable environment path. Please confirm that this is not the case, a simple 'which splunk' may show the executables outside of the current folder you're executing from. To cleanly check this, you can do a 'sudo killall splunkd && sudo killall mongod'. This will hard kill all Splunk processes on the system. And we can start this from scratch. You Splunk UF should be installed in /opt/splunkforwarder, or if you installed a HF it will be in /opt/splunk. If you installed this as systemd, you need to restart the process with a 'systemctl start Splunkd.service`. If you are running ./splunk start as a user, there are issues around this. So please confirm this. Also, please read through our docs on systemd. There are some nuances to how systemd works vs initd. See docs here : https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/RunSplunkassystemdservice#What_is_systemd.3F
... View more
