topic Re: Splunk not restart in Deployment Architecture

Splunk not restart

arun_kant_sharm — Wed, 30 Sep 2020 05:29:20 GMT

Hi experts,
I try to restart our splunk server, but its not start.

Earlier I try to start from UI, but it not start.
I also try to reboot if using CLI, but dont see any thing on console

I am using Splunk 7.2 in AWS EC2 instance (Amazon 1) , I am using splunk on that environment from last one year.

$SPLUNK_HOME/bin/splunk -version
$SPLUNK_HOME/bin/splunk -version
Splunk 7.2.6 (build c0bf0f679ce9)

uname -a
Linux abcdXyz 4.14.123-86.109.amzn1.x86_64 #1 SMP Mon Jun 10 19:44:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

strace /opt/splunk/bin/splunk start
execve("/opt/splunk/bin/splunk", ["/opt/splunk/bin/splunk", "start"], [/ 50 vars /]) = -1 ENOEXEC (Exec format error)
write(2, "strace: exec: Exec format error\n", 32strace: exec: Exec format error
) = 32
exit_group(1) = ?
+++ exited with 1 +++

Re: Splunk not restart

renjith_nair — Wed, 30 Sep 2020 05:29:25 GMT

@arun_kant_sharma ,

Its quite strange that you dont see anything in the console after the start command. Is the installation dir correct and are you able to see binaries there ?

Re: Splunk not restart

arun_kant_sharm — Thu, 21 May 2020 08:17:58 GMT

Yes Binaries are present in /opt/splunk/bin.

Re: Splunk not restart

PavelP — Wed, 30 Sep 2020 05:29:28 GMT

Hello @arun_kant_sharma

please try prepend strace to see more

strace /opt/splunk/bin/splunk start

Re: Splunk not restart

arun_kant_sharm — Thu, 21 May 2020 10:13:53 GMT

strace /opt/splunk/bin/splunk start
execve("/opt/splunk/bin/splunk", ["/opt/splunk/bin/splunk", "start"], [/* 50 vars */]) = -1 ENOEXEC (Exec format error)
write(2, "strace: exec: Exec format error\n", 32strace: exec: Exec format error
) = 32
exit_group(1) = ?
+++ exited with 1 +++

Re: Splunk not restart

PavelP — Wed, 30 Sep 2020 05:31:30 GMT

@arun_kant_sharma this error means your computer architecture is different than the splunk binary

What is your OS (uname -a, lsb_release) and what is the exact splunk version (x64, 86, arm)?

Re: Splunk not restart

arun_kant_sharm — Wed, 30 Sep 2020 05:29:39 GMT

I am using Splunk 7.2 in AWS EC2 instance (Amazon 1) , I am using splunk on that environment from last one year.

$SPLUNK_HOME/bin/splunk -version
Splunk 7.2.6 (build c0bf0f679ce9)

uname -a
Linux abcdXyz 4.14.123-86.109.amzn1.x86_64 #1 SMP Mon Jun 10 19:44:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Re: Splunk not restart

arun_kant_sharm — Fri, 22 May 2020 02:28:56 GMT

ll splunk*
-r-xr-xr-x 1 splunk splunk 0 May 21 04:13 splunk
-r-xr-xr-x 1 splunk splunk 49356952 Apr 11 2019 splunkd
-r-xr-xr-x 1 splunk splunk 465 Apr 11 2019 splunkdj
-r-xr-xr-x 1 splunk splunk 21904 Apr 11 2019 splunkmon
-r-xr-xr-x 1 splunk splunk 295008 Apr 11 2019 splunk-optimize
-r-xr-xr-x 1 splunk splunk 291136 Apr 11 2019 splunk-optimize-lex

I don't know why my env splunk binary deleted, I only try to restart from UI. After replacing it from the other env, its working fine.

Re: Splunk not restart

ayush1906 — Fri, 22 May 2020 02:48:51 GMT

Hi Arun,

You are logged in as root user, and that does not have access to restart splunk.
Either do -> sudo su - splunk , then give restart command

or use chown command to change the owner to splunk then it will surely work.

Kindly accept this as answer if it works for you 🙂

Re: Splunk not restart

renjith_nair — Wed, 30 Sep 2020 05:29:41 GMT

@arun_kant_sharma ,
its possible that the binaries are overwritten by manual copy/move process. Otherwise it should output the start up messages in your console

Re: Splunk not restart

PavelP — Fri, 22 May 2020 06:03:36 GMT

Has splunk suddently stopped to work or it happened after an upgrade?

please try

file /opt/splunk/bin/splunk*

expected output:

[root@mwg42 ~]# file /opt/splunk/bin/splunk*
/opt/splunk/bin/splunk:              ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped
/opt/splunk/bin/splunkd:             ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped
/opt/splunk/bin/splunkmon:           ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped
/opt/splunk/bin/splunk-optimize:     ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped
/opt/splunk/bin/splunk-optimize-lex: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped