AWS MarketPlace SPLUNK instance type upgrade

[lorezyra]

I created an instance from the marketplace and later decided to change the instance type.
When I stopped the instance, I know that AWS will delete the ephemeral storage. I never configured SPLUNK to use it.
However, after the instance upgrade (from c5d.2xlarge to c5d.4xlarge), the SPLUNKd service fails to start.

What are the next recommended steps to get SPLUNK running again?
I'm not finding anything useful in /opt/splunk/var/log/splunk/splunkd.log ...

[nickhills · Accepted solution]

if you run /opt/splunk/bin/splunk restart on the CLI does it give you any information about the startup progress. Do you see any errors?

Did you originally run Splunk enable boot-start.

Finally, what version of Splunk, and is it systemd?

[lorezyra]

Using MarketPlace SPLUNK Enterprise (core) 7.2.0

Splunk status initially spat error about splunkd not running and hung on killing the child processes. After rebooting the server, splunk status then complained about permission denied for the config files.
Found the config files were set to root:root. So, I chown splunk:splunk -R /opt/splunkand reboot the server again. Then everything came back up!

So, my question now: Why did I need to do all that?

[nickhills]

If you ran Splunk as root by mistake, it may have changed the permissions on the folders, then later when you run it as the Splunk user, it can’t access the files.

It’s easily done.