Why is splunkd not running after downgrading from 8.3.3 to 8.1.6?

[tokio13]

Hello,

I'm encountering the following issue on one of my indexers (from a total of 3) after downgrading from 8.3.3 to 8.1.6. All my other components (3SH,CM,MC,Deployer, Indexer2 and 3 are working fine after downgrading.)

I tried pretty much everything to kill the process, restart splunk, restart the instance on the cloud, nothing seems to help.

splunkd is not running.

[tokio13]

What worked was, removing it and installing 8.1.6 all over again as you recommended. 

Thanks!

[isoutamo]

Great, if needed there are some discussion about rebalance events https://community.splunk.com/t5/Installation/New-indexer-sync-is-very-slow/m-p/538596

[SanjayReddy]

Hi @tokio13 

what  error message are  you seeing while starting the splunkd  and Splunkd.log of Indexer?.

any dubplicate bukctet confilit in Splunkd.log issues ? 

any lock file present under /var/run/splunk/

was there any permission issue for directrory ? 

if you clould share the error logs , that can help to troubleshoot furthur 

[isoutamo · Accepted solution]

Splunk is not supporting downgrade after update very well 😞 For that reason the instructions are do a backup and restore it as a downgrade, but then you of course lost some events.

What this instance has written to it's splunkd.log on filesystem? There should be reason why it didn't start.

As you have an indexer cluster and if/when your SF+RF >=2 then maybe the easiest way is just remove that instance from cluster and reinstall it with 8.1.6 and then rejoin it back? Of course it needs that your CM said that you still have all data searchable.

r. Ismo