How to know if there are custom changes made to an App from btools log?

[zacksoft_wf]

I am checking the upgrade readiness of my Splunk Apps under ES search-head.  (we have SearchHead cluster and changes are pushed from there, and I am told the changed once pushed flattens both local and default folder)
While checking I also want to know if there are any custom changes made to any app. If any custom changes I need to inform the respective owner to fix it before updating the app to make it py2 and py3 compatible.

Now, I don't have access to any file system. I am told by looking at btool debug logs, I can find out if any custom changes are applied to any app. I just don't know what specific in btool log will give it away if any custom change is done ? What exactly to look for in btool to know that ?

Any  assistance here ?

[PickleRick]

There are two sides to this question.

One is what you mean by "don't have access to any file system". Without access to the operating system you won't be able to get the output from the btool. That's one thing.

Another thing is that even with btool you can - as @tshah-splunk said fairly easily spot changes to the default app settings but there are two issues with that. But that doesn't mean that the app wasn't modify on the deployer in the first place, especially thay you're saying that the apps are "flattened" upon distribution which means the deployer most probably works in the "merge to default" mode.

And even if you can spot changes entered in local/ folders on the deployer, you have absolutely no "technical" guarantee that someone didn't make changes in the default/ directories. You shouldn't do that but people sometimes do.

So short of comparing the app state with a "vanilla" app freshly decompressed from the package you have no way to be sure. That's why it's useful to have some form of configuration tracking.

[zacksoft_wf]

I meant I can't open the file system in the standard way by using a command prompt and typing  btool debug command.  Lets say I want to see if there is any custom override for the TA Symantec.

I write this in the splunk search bar to get the btool info.  
index=splunk_admin sourcetype="an:1SPLK:config_file_btool_output_debug:txt" host="aws3e-bullcp0135.anz.net" "/opt/splunk/etc/apps/Splunk_TA_symantec-ep"

and I see output like,
/opt/splunk/etc/apps/Splunk_TA_symantec-ep/default/transforms.conf [symantec_ep_severity_lookup] /opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True /opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True /opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE = /opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/system/default/transforms.conf DEST_KEY = /opt/splunk/etc/system/default/transforms.conf FORMAT = /opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False /opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096 /opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/transforms.conf MV_ADD = False /opt/splunk/etc/system/default/transforms.conf REGEX = /opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw 

Just because there is no etc/system/local   above and all are etc/system/default
Can I say that there is no custom changes done to Symantec app?

Note : my admin said something like, "all our apps are installed in ES(Enterprise Security) search head. and when updated are pushed from deployer it flattens both default as wells as local."  
(I don't exactly understand what he meant.)

[PickleRick · Accepted solution]

I can't know for sure what your admin meant 😉 But the most probable thing is that indeed your deployer uses "merge to default" deployment scheme which means that even if you have some settings in both - for example - default/inputs.conf and local/inputs.conf during deployment they will be merged and stored in default/inputs.conf on the destination search head.

So now you have three possible scenarios

1) A setting comes from the default configuration of an app as unpacked on the deployer. The setting is therefore placed in the default/ directory on the deployer and is deployed to the default/ directory on search heads

2) A settings is overwritten with local/ directory on the deployer. During deployment it's getting merged and placed in the default/ directory on the search head (along those default settings from previous point)

3) A setting is manually overwritten on the search head (usually by changing something using UI) - it's getting placed in local/ directory on the search head (and usually distributed among all search heads in the cluster; unless the replication configuration has been deliberately changed)

There is also always the possibility that a setting from p.1 above has been modified beforehand and even though it's placed in the default/ directory it's been tampered with.

And there is one more thing - btool will show you a resulting config file from merging all files according to the precedence but it doesn't mean that the setting from the local/ directory is different than the default one.

So if you have - again let's use inputs.conf as example - default/inputs.conf listing a stanza with "disabled=true" setting and you have local/inputs.conf again with the same "disabled=true", you will get the local/inputs.conf as the source of this setting even though it doesn't differ from the default one.

So just using one tool it's impossible to do a full verification of your config state.

[tshah-splunk]

Hey @zacksoft_wf,

When you run the btool command with debug mode enabled, look out for the file paths that are having local directory in them. All the changes made to the local directory would be the custom changes made as per the requirement. Basic syntax for btool command will be like below:

$SPLUNK_HOME/bin/splunk btool <<file_name>> list --debug

PS: Do not add the .conf extension when you add a file.