Why the error "Can't create directory" when running "splunk show config"?

[vzabawski]

Hello. I'm trying to view Splunk configuration, but getting a very odd error:

splunk@test1:/> /opt/splunk/bin/splunk show config authentication
Splunk username: admin
Password:
Can't create directory "/opt/splunk/splunk/.splunk": No such file or directory

 Changing dir to /opt/splunk/bin/ and running command from it also doen't help.

Doesn't matter if I run the command under splunk user (this user owns files in /opt/splunk) or with sudo.

This Splunk instance was upgraded from 7.0.0 to 8.1.5 and then to 8.2.5, so maybe it affected somehow.

How can I fix this?

[vzabawski · Accepted solution]

This issue is caused by wrong home directory set for "splunk" user. I've compared the output of echo $HOME on healthy and this server I'm having troubles with and on "healthy" server HOME is set to /opt/splunk.

On affected server HOME is set to /opt/splunk/splunk, so that's what was causing problems.

[Stefanie]

It looks like it thinks your splunk environment was installed in /opt/splunk/splunk ...

Can you check to see what your $SPLUNK_HOME variable is set to? 
If you open /opt/splunk/etc/splunk-launch.conf do you see SPLUNK_HOME=/opt/splunk?

[vzabawski]

Good idea, thanks! SPLUNK_HOME looks fine, so I guess it's not the case.

[Stefanie]

I'd reinstall 8.2.5 on top of it and see if it would resolve it? Is this the only server that's affected? 

Just to make sure.. can you cd to 

/opt/splunk/.splunk

Just to see if that directory exists.

What happens when you do the command 

cd $SPLUNK_HOME

Does it take you to /opt/splunk or does it throw an error for /opt/splunk/splunk