Monitoring Splunk

Error message when creating new Splunk instance: The splunk daemon (splunkd) is already running. [FAILED]

jackiewkc
Path Finder

Hi,

I am running a splunk instance on a server under /apps/splunk-1/ at port 8980. I would like to run another instance on the same server at a different port. So I ran "cp -R /app/splunk-1 /apps/splunk-2" to create a new instance. Then I removed the pid file under /apps/splunk-2/var/run, updated the related files so that it will use a different port (8950).

However, when I ran "/apps/splunk-2/bin/splunk start", I got the following error:

The splunk daemon (splunkd) is already running. [FAILED]

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://server1:8980.

When I executed /apps/splunk-2/bin/splunk status, I got this:

splunkd is running (PID: 3898).
splunk helpers are running (PIDs: 3903 4024 4060 4083).

Can someone please advise why splunk-2 is still referencing splunk-1?

What else do I need to update under splunk-2?

Thanks.

Regards,
Jackie

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

Check the following configuration files, they should be located in the /app/splunk-2/etc/system/local :

instance.cfg : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Instancecfgconf
web.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Webconf
inputs.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Inputsconf
server.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Serverconf

You'll need to update the configurations, change the ports, GUID, instance name, and any other specific configuration items you might have copied over. After that, I'd run a killall splunkd and killall mongod to make sure all the processes are killed. From there, you can run /opt/splunk-1/bin/splunk start and /opt/splunk-2/bin/splunk start.

You should be careful about the run-as user and permissions associated with each running Splunk instance....

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Check the following configuration files, they should be located in the /app/splunk-2/etc/system/local :

instance.cfg : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Instancecfgconf
web.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Webconf
inputs.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Inputsconf
server.conf : http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Serverconf

You'll need to update the configurations, change the ports, GUID, instance name, and any other specific configuration items you might have copied over. After that, I'd run a killall splunkd and killall mongod to make sure all the processes are killed. From there, you can run /opt/splunk-1/bin/splunk start and /opt/splunk-2/bin/splunk start.

You should be careful about the run-as user and permissions associated with each running Splunk instance....

0 Karma

jackiewkc
Path Finder

Hi, thanks for the information. I have updated all those files already. Under /apps/splunk-2/etc/system/local, I ran "grep -ir 8980 ." to confirm that there is no reference to the port used by splunk-1.

Can you please advise how ./splunk start and ./splunk status work? i.e. where do they look for the related configs?

/apps/splunk-2/bin/splunk status still references splunk-1 so it must look at somewhere to get the port, pid file, the name of the instance or something else to check the status, right?

Regards,
Jackie

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

How are you starting and stopping Splunk?

Additional, check out your splunk-launch.conf file : https://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Splunk-launchconf

Make sure your referrences are in there properly. These are configured in the initial installation.

Alternatively, you should just install a fresh copy of Splunk in /opt/splunk-2. Download the tarball and extract it to there.

0 Karma

jackiewkc
Path Finder

Hi, splunk-launch.conf did the trick, thanks for your help.

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...