We noticed we have 2 different processes running:
systemctl status splunk
systemctl status Splunkd
The docs explain that there are 2 ways to setup the Splunk service
https://docs.splunk.com/Documentation/Splunk/latest/Admin/RunSplunkassystemdservice#Unit_file_naming...
Is it correct that the 2 are running in parallel on the same machine? should it not be either one or the other?
This tells me that you have two Splunk process running under different users.
The daemon name is configured withing /opt/splunk/etc/splunk-launch.conf
You likely have one running from init.d (default) and another from systemd after configuring it.
Assuming this is not production, I would suggest that you try the following, in order:
/opt/splunk/bin/splunk stop
systemctl stop Splunkd
(here is where you can modify the daemon name in /opt/splunk/etc/splunk-launch.conf)
systemctl start Splunkd (or the name you set within the conf file)