After Splunk forwarder upgrade to version 7.3.0 from 6.6.x - splunk forwarder is not starting ?

[rakesh_498115]

Hi All,

After upgrading my splunk forwarder to version 7.3.0 from 6.6.x. my splunk forwarder didnt start. it is shwoing the below error.

A Splunk installation already exists. This will upgrade the current installation.
Do you still wish to continue ?: [y|n]
y
Continuing with update
Check for processes...

Extracting 'splunkforwarder-7.3.0-657388c7a488-Linux-x86_64.gz' ...
Updating config files...
Starting the forwarder...
install_nix_forwarder.sh: line 199: /opt/splunkforwarder/bin/splunk: Operation not permitted
Splunk did not start. Please check any error messages

when checked in error logs i couldnt find much information -

09-04-2019 19:34:54.397 +1000 INFO TcpOutputProc - Connected to idx=10.16.193.244:9997, pset=0, reuse=0. using ACK.
09-04-2019 19:35:27.370 +1000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.16.195.198_8089_10.16.195.198_lnxau2106st0273.wsdc.nsw.westpac.com.au_Splunk_Forwarder_payments_lnxau2106st0273
09-04-2019 19:35:30.813 +1000 WARN TcpOutputProc - Cooked connection to ip=10.16.193.247:9997 timed out
09-04-2019 19:35:50.653 +1000 WARN TcpOutputProc - Cooked connection to ip=10.17.193.39:9997 timed out
09-04-2019 19:35:50.759 +1000 INFO TcpOutputProc - Connected to idx=10.17.193.38:9997, pset=0, reuse=0. using ACK.
09-04-2019 19:36:05.484 +1000 INFO PipelineComponent - Performing early shutdown tasks
09-04-2019 19:36:05.503 +1000 INFO loader - Shutdown HTTPDispatchThread
09-04-2019 19:36:05.514 +1000 INFO ShutdownHandler - Shutting down splunkd
09-04-2019 19:36:05.514 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_Begin"
09-04-2019 19:36:05.531 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_FileIntegrityChecker"
09-04-2019 19:36:05.531 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_JustBeforeKVStore"
09-04-2019 19:36:05.531 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_KVStore"
09-04-2019 19:36:05.531 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_Thruput"
09-04-2019 19:36:05.531 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput1"
09-04-2019 19:36:05.540 +1000 INFO TcpInputProc - Running shutdown level 1. Closing listening ports.
09-04-2019 19:36:05.541 +1000 INFO TcpInputProc - Shutting down listening ports
09-04-2019 19:36:05.542 +1000 INFO TcpInputProc - Setting up input quiesce timeout for : 90.000 secs
09-04-2019 19:36:06.335 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_ExecSendInitialSigterm"
09-04-2019 19:36:06.335 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput"
09-04-2019 19:36:06.335 +1000 INFO TcpOutputProc - begin to shut down auto load balanced connection strategy
09-04-2019 19:36:06.336 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_UdpInput"
09-04-2019 19:36:06.336 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_FifoInput"
09-04-2019 19:36:06.336 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_WinEventLogInput"
09-04-2019 19:36:06.336 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpInput"
09-04-2019 19:36:06.338 +1000 INFO TcpInputProc - Cleaning up TCP connections
09-04-2019 19:36:06.338 +1000 INFO TcpInputProc - Shutting down existing connections.
09-04-2019 19:36:06.339 +1000 INFO TcpInputProc - TCP connection cleanup complete
09-04-2019 19:36:06.349 +1000 INFO ShutdownHandler - shutting down level "ShutdownLevel_CacheManager"

Could someone please help to diagnose the problem and fix it.

[burakcinar]

hi rakesh_498115,

could you try delete splunkd.pif file and start splunk again? its under "$SPLUNK_HOME/var/run/splunk/"

export SPLUNK_HOME="/opt/splunk"
rm $SPLUNK_HOME/var/run/splunk/splunkd.pid
$SPLUNK_HOME/bin/splunk status
$SPLUNK_HOME/bin/splunk start

another option is check splunk user has right to splunk home.

chown -R splunk:splunk /opt/splunk/

And start splunk with "splunk" user.

if problem stills exists could you share content of install_nix_forwarder.sh file ?