I have this kind of weird custom app (and dangerous too) that changes the UF Instance GUID. Basically, I created a .sh file, which utilizes "sed" command on Linux, to change the UUID value of the /opt/splunkforwarder/etc/instance.cfg file.
To use a .sh script and make changes to SPLUNK_HOME directory is quite a dangerous task, I advised not to, however, this task is quite simple, I tested so I decided to deploy an app called REGEN_GUID with a single inputs.conf file that have the stanza to run the script.
[script://./bin/regenerate_guid.sh]
interval = 900
source = regenerate_guid
sourcetype = regenerate_guid
index = <REDACTED>
disabled = 0
In general, quite simple, and it run. I could change the instance UUID and nothing critical happened. However, of course after I see that the UUID has been changed, I would remove the client from the app. I used the deployment server UI, go into the app section and remove the IP of the instance from the whitelist.
Checking the splunkd.log, I could see the log when it say it is removing the app
However, after that, I check again and see the log and see it is still finding the script to run, the log appear every 15 minutes, which is equal to the script interval, so basically the UF is still interpreting the task of running the script. The log is like this:
05-07-2025 11:00:07.938 +0700 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/REGEN_GUID/bin/regenerate_guid.sh" /bin/sh: 1: /opt/splunkforwarder/etc/apps/REGEN_GUID/bin/regenerate_guid.sh: not found
Does anyone know the reason? I think the reason might be the way Splunk monitor script inputs is through some kinds of cron file, and my app failed to update that when it was removed?
