Monitoring Splunk

Error message when creating new Splunk instance: The splunk daemon (splunkd) is already running. [FAILED]

jackiewkc
Path Finder

Hi,

I am running a splunk instance on a server under /apps/splunk-1/ at port 8980. I would like to run another instance on the same server at a different port. So I ran "cp -R /app/splunk-1 /apps/splunk-2" to create a new instance. Then I removed the pid file under /apps/splunk-2/var/run, updated the related files so that it will use a different port (8950).

However, when I ran "/apps/splunk-2/bin/splunk start", I got the following error:

The splunk daemon (splunkd) is already running. [FAILED]

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://server1:8980.

When I executed /apps/splunk-2/bin/splunk status, I got this:

splunkd is running (PID: 3898).
splunk helpers are running (PIDs: 3903 4024 4060 4083).

Can someone please advise why splunk-2 is still referencing splunk-1?

What else do I need to update under splunk-2?

Thanks.

Regards,
Jackie

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

Check the following configuration files, they should be located in the /app/splunk-2/etc/system/local :

instance.cfg : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Instancecfgconf
web.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Webconf
inputs.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Inputsconf
server.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Serverconf

You'll need to update the configurations, change the ports, GUID, instance name, and any other specific configuration items you might have copied over. After that, I'd run a killall splunkd and killall mongod to make sure all the processes are killed. From there, you can run /opt/splunk-1/bin/splunk start and /opt/splunk-2/bin/splunk start.

You should be careful about the run-as user and permissions associated with each running Splunk instance....

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Check the following configuration files, they should be located in the /app/splunk-2/etc/system/local :

instance.cfg : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Instancecfgconf
web.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Webconf
inputs.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Inputsconf
server.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Serverconf

You'll need to update the configurations, change the ports, GUID, instance name, and any other specific configuration items you might have copied over. After that, I'd run a killall splunkd and killall mongod to make sure all the processes are killed. From there, you can run /opt/splunk-1/bin/splunk start and /opt/splunk-2/bin/splunk start.

You should be careful about the run-as user and permissions associated with each running Splunk instance....

0 Karma

jackiewkc
Path Finder

Hi, thanks for the information. I have updated all those files already. Under /apps/splunk-2/etc/system/local, I ran "grep -ir 8980 ." to confirm that there is no reference to the port used by splunk-1.

Can you please advise how ./splunk start and ./splunk status work? i.e. where do they look for the related configs?

/apps/splunk-2/bin/splunk status still references splunk-1 so it must look at somewhere to get the port, pid file, the name of the instance or something else to check the status, right?

Regards,
Jackie

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

How are you starting and stopping Splunk?

Additional, check out your splunk-launch.conf file : https://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Splunk-launchconf

Make sure your referrences are in there properly. These are configured in the initial installation.

Alternatively, you should just install a fresh copy of Splunk in /opt/splunk-2. Download the tarball and extract it to there.

0 Karma

jackiewkc
Path Finder

Hi, splunk-launch.conf did the trick, thanks for your help.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...