Error message when creating new Splunk instance: The splunk daemon (splunkd) is already running. [FAILED]

[jackiewkc]

Hi,

I am running a splunk instance on a server under /apps/splunk-1/ at port 8980. I would like to run another instance on the same server at a different port. So I ran "cp -R /app/splunk-1 /apps/splunk-2" to create a new instance. Then I removed the pid file under /apps/splunk-2/var/run, updated the related files so that it will use a different port (8950).

However, when I ran "/apps/splunk-2/bin/splunk start", I got the following error:

The splunk daemon (splunkd) is already running. [FAILED]

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://server1:8980.

When I executed /apps/splunk-2/bin/splunk status, I got this:

splunkd is running (PID: 3898).
splunk helpers are running (PIDs: 3903 4024 4060 4083).

Can someone please advise why splunk-2 is still referencing splunk-1?

What else do I need to update under splunk-2?

Thanks.

Regards,
Jackie

[esix_splunk · Accepted solution]

Check the following configuration files, they should be located in the /app/splunk-2/etc/system/local :

instance.cfg : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Instancecfgconf
web.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Webconf
inputs.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Inputsconf
server.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Serverconf

You'll need to update the configurations, change the ports, GUID, instance name, and any other specific configuration items you might have copied over. After that, I'd run a killall splunkd and killall mongod to make sure all the processes are killed. From there, you can run /opt/splunk-1/bin/splunk start and /opt/splunk-2/bin/splunk start.

You should be careful about the run-as user and permissions associated with each running Splunk instance....

[jackiewkc]

Hi, thanks for the information. I have updated all those files already. Under /apps/splunk-2/etc/system/local, I ran "grep -ir 8980 ." to confirm that there is no reference to the port used by splunk-1.

Can you please advise how ./splunk start and ./splunk status work? i.e. where do they look for the related configs?

/apps/splunk-2/bin/splunk status still references splunk-1 so it must look at somewhere to get the port, pid file, the name of the instance or something else to check the status, right?

Regards,
Jackie

[esix_splunk]

How are you starting and stopping Splunk?

Additional, check out your splunk-launch.conf file : https://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Splunk-launchconf

Make sure your referrences are in there properly. These are configured in the initial installation.

Alternatively, you should just install a fresh copy of Splunk in /opt/splunk-2. Download the tarball and extract it to there.

[jackiewkc]

Hi, splunk-launch.conf did the trick, thanks for your help.