Indexer Splunkd services are not able to run

[phanichintha]

In indexer cluster environment one of the Indexer got stopped unable to start/restart
C:\Windows\system32>d:
D:>cd spluk\bin
The system cannot find the path specified.
D:>cd splunk\bin
D:\Splunk\bin>.\splunk restart
Splunkd: Stopped
Splunk> All batbelt. No tights.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
(skipping validation of index paths because not running as
LocalSystem)
Validated: _audit _internal _introspection _telemetry _thef
ishbucket aws_anomaly_detection aws_topology_daily_snapshot aws_topology_hi
story aws_topology_monthly_snapshot aws_topology_playback aws_vpc_flow_logs
history main summary
Done
Bypassing local license checks since this instance is configured with a rem
ote license master.
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from 'D:\Splunk\splunk-7.
2.1-be11b2c46e23-windows-64-manifest'
All installed files intact.
Done
Checking replication_port port [7778]: open
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Splunkd: Starting (pid 6420)
Timed out waiting for splunkd to start.

Splunkd.log
05-18-2020 07:31:58.157 +0000 INFO ServerRoles - Declared role=cluster_slave.
05-18-2020 07:31:58.157 +0000 INFO ServerRoles - Declared role=indexer.
05-18-2020 07:31:58.157 +0000 INFO ClusteringMgr - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=60.000 rst=60.000 rrt=60.000 rmst=180.000 rmrt=180.000 icps=-1 sfrt=600.000 pe=1 im=0 is=1 mob=5 mor=5 mosr=5 pb=5 rep_port=port=7778 isSsl=0 ipv6=0 cipherSuite= ecdhCurveNames= sslVersions=SSL3,TLS1.0,TLS1.1,TLS1.2 compressed=1 allowSslRenegotiation=1 dhFile= reqCliCert=0 serverCert= rootCA= commonNames= alternateNames= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=true allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1
05-18-2020 07:31:58.172 +0000 INFO ClusteringMgr - Initializing node as slave
05-18-2020 07:31:58.172 +0000 INFO BucketReplicator - Initializing BucketReplicatorMgr
05-18-2020 07:31:58.219 +0000 INFO CMServiceThread - CMHealthManager starting eloop
05-18-2020 07:31:58.235 +0000 INFO CMBundleMgr - bundle=D:\Splunk\var\run\splunk\cluster\remote-bundle\2df598296706d9846433003de4c7a927-1589221919.bundle, checksum=5F5C9F53A58CD618B69209EBC5D92286 found on the slave
05-18-2020 07:31:58.235 +0000 INFO CMBundleMgr - setting active bundle= to latest bundle=6F0874F9DA123EA345D25A77F6D3CAFA
05-18-2020 07:31:58.235 +0000 INFO CMSlave - event=getActiveBundle status=success path=D:\Splunk\var\run\splunk\cluster\remote-bundle\83209f7543173582062b08f2b77fcde0-1589259155.bundle cksum=6F0874F9DA123EA345D25A77F6D3CAFA alreadyin=0
05-18-2020 07:31:58.235 +0000 ERROR CMSlave - event=move downloaded bundle to slave-apps failed with err="failed to remove dir=D:\Splunk\etc\slave-apps.old (There are no more files.)" even after multiple attempts, Exiting..
05-18-2020 07:31:58.235 +0000 ERROR loader - Failed to download bundle from master, err="failed to remove dir=D:\Splunk\etc\slave-apps.old (There are no more files.)", Won't start splunkd.

please provide the solution if any one knows.

[PavelP · Accepted solution]

Hello @phanichintha,

as there are no splunkd.log provided as asked by @richgalloway , you'd be better to open a support ticket

[phanichintha]

Hello guys, pl check this
Splunkd.log
05-18-2020 07:31:58.157 +0000 INFO ServerRoles - Declared role=cluster_slave.
05-18-2020 07:31:58.157 +0000 INFO ServerRoles - Declared role=indexer.
05-18-2020 07:31:58.157 +0000 INFO ClusteringMgr - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=60.000 rst=60.000 rrt=60.000 rmst=180.000 rmrt=180.000 icps=-1 sfrt=600.000 pe=1 im=0 is=1 mob=5 mor=5 mosr=5 pb=5 rep_port=port=7778 isSsl=0 ipv6=0 cipherSuite= ecdhCurveNames= sslVersions=SSL3,TLS1.0,TLS1.1,TLS1.2 compressed=1 allowSslRenegotiation=1 dhFile= reqCliCert=0 serverCert= rootCA= commonNames= alternateNames= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=true allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1
05-18-2020 07:31:58.172 +0000 INFO ClusteringMgr - Initializing node as slave
05-18-2020 07:31:58.172 +0000 INFO BucketReplicator - Initializing BucketReplicatorMgr
05-18-2020 07:31:58.219 +0000 INFO CMServiceThread - CMHealthManager starting eloop
05-18-2020 07:31:58.235 +0000 INFO CMBundleMgr - bundle=D:\Splunk\var\run\splunk\cluster\remote-bundle\2df598296706d9846433003de4c7a927-1589221919.bundle, checksum=5F5C9F53A58CD618B69209EBC5D92286 found on the slave
05-18-2020 07:31:58.235 +0000 INFO CMBundleMgr - setting active bundle= to latest bundle=6F0874F9DA123EA345D25A77F6D3CAFA
05-18-2020 07:31:58.235 +0000 INFO CMSlave - event=getActiveBundle status=success path=D:\Splunk\var\run\splunk\cluster\remote-bundle\83209f7543173582062b08f2b77fcde0-1589259155.bundle cksum=6F0874F9DA123EA345D25A77F6D3CAFA alreadyin=0
05-18-2020 07:31:58.235 +0000 ERROR CMSlave - event=move downloaded bundle to slave-apps failed with err="failed to remove dir=D:\Splunk\etc\slave-apps.old (There are no more files.)" even after multiple attempts, Exiting..
05-18-2020 07:31:58.235 +0000 ERROR loader - Failed to download bundle from master, err="failed to remove dir=D:\Splunk\etc\slave-apps.old (There are no more files.)", Won't start splunkd.

[PavelP]

Hello @phanichintha

ERROR loader - Failed to download bundle from master, err="failed to remove dir=D:\Splunk\etc\slave-apps.old (There are no more files.)", Won't start splunkd.

remove D:\Splunk\etc\slave-apps.old folder and try again

[phanichintha]

Hello PaveIP thank you so much for your answer, after removed D:\Splunk\etc\slave-apps.old its restared.

[phanichintha]

PavelP i have another question actually i stuck with something, can you please check if you have an idea about this.
https://answers.splunk.com/answers/821635/splunk-add-on-for-unix-and-linux-pssh-kafka-logs-a.html

[PavelP]

I'll check it, Please accept the previous answer if it solved your query.

[phanichintha]

How to accept answer here, i didn't see any popup. can you help out.

[PavelP]

please press "accept " link, it is located just after my answer in the same line with "Add comment · award points · accept". Thank you.

[richgalloway]

Have you checked splunkd.log?

[richgalloway]

Have you checked splunkd.log on the indexer?

[phanichintha]

05-14-2020 05:12:43.575 +0000 INFO ServerRoles - Declared role=cluster_slave.
05-14-2020 05:12:43.575 +0000 INFO ServerRoles - Declared role=indexer.
05-14-2020 05:12:43.575 +0000 INFO ClusteringMgr - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=60.000 rst=60.000 rrt=60.000 rmst=180.000 rmrt=180.000 icps=-1 sfrt=600.000 pe=1 im=0 is=1 mob=5 mor=5 mosr=5 pb=5 rep_port=port=7778 isSsl=0 ipv6=0 cipherSuite= ecdhCurveNames= sslVersions=SSL3,TLS1.0,TLS1.1,TLS1.2 compressed=1 allowSslRenegotiation=1 dhFile= reqCliCert=0 serverCert= rootCA= commonNames= alternateNames= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=true allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1
05-14-2020 05:12:43.575 +0000 INFO ClusteringMgr - Initializing node as slave
05-14-2020 05:12:43.575 +0000 INFO BucketReplicator - Initializing BucketReplicatorMgr
05-14-2020 05:12:43.638 +0000 INFO CMServiceThread - CMHealthManager starting eloop
05-14-2020 05:12:43.638 +0000 INFO CMBundleMgr - bundle=D:\Splunk\var\run\splunk\cluster\remote-bundle\2df598296706d9846433003de4c7a927-1589221919.bundle, checksum=5F5C9F53A58CD618B69209EBC5D92286 found on the slave
05-14-2020 05:12:43.638 +0000 INFO CMBundleMgr - setting active bundle= to latest bundle=6F0874F9DA123EA345D25A77F6D3CAFA
05-14-2020 05:12:43.638 +0000 INFO CMSlave - event=getActiveBundle status=success path=D:\Splunk\var\run\splunk\cluster\remote-bundle\83209f7543173582062b08f2b77fcde0-1589259155.bundle cksum=6F0874F9DA123EA345D25A77F6D3CAFA alreadyin=0
05-14-2020 05:12:43.638 +0000 ERROR CMSlave - event=move downloaded bundle to slave-apps failed with err="failed to remove dir=D:\Splunk\etc\slave-apps.old (There are no more files.)" even after multiple attempts, Exiting..
05-14-2020 05:12:43.638 +0000 ERROR loader - Failed to download bundle from master, err="failed to remove dir=D:\Splunk\etc\slave-apps.old (There are no more files.)", Won't start splunkd.