Error while deploying apps to first member [Splunk Enterprise 8.2]

[Ciccius]

Hi all,
I am trying to deploy my apps from the deployment server with the command: 

/opt/splunk/bin/splunk apply shcluster-bundle -target https://splunksrc:8089 -preserve-lookups true

It never failed to do the task but now I am getting this error:

Error while deploying apps to first member, aborting apps deployment to all members: Error while deleting app=rest_ta on target=https://splunksrc:8089: Non-200/201 status_code=500; {"messages":[{"type":"ERROR","text":"\n In handler 'localapps': Cannot update application info: /nobody/rest_ta/app/install/state = disabled: Could not find writer for: /nobody/rest_ta/app/install/state [0] [/opt/splunk/etc]"}]}

Both the nodes (deployment and splunksrc) have enough disk space.

Any ideas?

Thanks
Francesco

[livehybrid]

Hi @Ciccius 

This is likely to be an issue with permissions, please could you validate that the permissions within the rest_ta app in $SPLUNK_HOME/etc/apps/rest_ta is the same across your SHC? 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[asimit · Accepted solution]

Hi @Ciccius 

Based on the error message you're receiving, this appears to be a permissions issue with the rest_ta app. The specific error "Could not find writer for: /nobody/rest_ta/app/install/state" suggests that Splunk doesn't have the proper permissions to update the app's state.

## Troubleshooting steps:

1. **Check permissions on the app directory**:
```
sudo ls -la /opt/splunk/etc/apps/rest_ta/
```
Make sure the directory and files are owned by the Splunk user and group.

2. **Fix permissions if needed**:
```
sudo chown -R splunk:splunk /opt/splunk/etc/apps/rest_ta/
sudo chmod -R 755 /opt/splunk/etc/apps/rest_ta/
```

3. **Try disabling the app manually before deployment**:
- On the deployment server, edit `/opt/splunk/etc/apps/rest_ta/default/app.conf`
- Set `state = disabled` in the `[install]` section
- Or completely remove the app if it's not needed: `sudo rm -rf /opt/splunk/etc/apps/rest_ta/`

4. **Check for file system issues**:
- The error might indicate file system corruption or disk issues
- Run `df -h` to check disk space (you mentioned this is fine)
- Run `sudo touch /opt/splunk/etc/test.txt` to verify write permissions to the directory

5. **Validate the deployment server's configuration**:
```
sudo /opt/splunk/bin/splunk show shcluster-bundle-status
```

6. **Restart Splunk on both servers**:
```
sudo /opt/splunk/bin/splunk restart
```

7. **Deploy without the problematic app**:
- Temporarily move the app out of the deployment directory
- Try the deployment again
- If successful, the issue is definitely with the app itself

If the issue persists, you may need to check Splunk logs for more details:
```
sudo cat /opt/splunk/var/log/splunk/splunkd.log | grep rest_ta
```

Let me know if any of these steps help resolve the issue!

[Ciccius]

Hi asimit,

thank you very much, it was a permission issue.

I don't know why the user/group for app rest_ta was root/root, once I reset to splunk/splunk it worked.

Thanks!