Getting Data In

Simple Configuration/File Precedence Not Working

Path Finder

Hi everyone,

I have SplunkTAaws and SplunkTAawsknowledgeonly deployed to a distributed splunk environment from a deployment server. On the heavy forwarders UI we have configured inputs which get placed locally on the heavy forwarders in SplunkTAaws/local/inputs.conf. Now I need to adjust some search time extractions so on the deployment server I have edited SplunkTAawsknowledgeonly/local/props.conf with my changes. This is so when I deploy config the local inputs on the heavy forwarders are not overwritten. It is my understanding that SplunkTAawsknowledgeonly/local/props.conf should be able to overwrite SplunkTAaws/default/props.conf given that they contain the same stanza name but my search time extractions don't seem to be working. If I however simply move the file from SplunkTAawsknowledgeonly/local/props.conf to SplunkTAaws/local/props.conf manually on a test server the extractions begin working immediately.

What am I doing wrong?

0 Karma

Path Finder

I have been working with support for a little while now. So far we have determined that it is indeed tied to the name of the app somehow. If I rename SplunkTAawsknowledgeonly/ to just knowledgeonly/ then the local/props in knowledgeonly/ overrides the default/props in SplunkTA_aws/ as one would expect.

We believe it has something to do with app vs global context as outlined here:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Admin/Wheretofindtheconfigurationfiles
(scroll down to "Summary of the effect of directories on configuration precedence")

We also tried setting the props stanza priority manually via:
priority = 5
where the priority on local is manually set to a higher number and default is manuallyl set to a lower number.
This had no effect when using the folders SplunkTAaws/default and SplunkTAaws_knowledgeonly/local.

At this point I can deploy the config I wanted to by just using a different app name and I am confident that something specific with this TA is going on. I have many other apps with the same name prefix just like this which function fine. Support will continue to investigate but at this point I am stumped.

0 Karma

Esteemed Legend

The one app "that works" has a default.meta file that exports permissions to All apps AKA global whereas the other file does not. Copy the default.meta file from the working app to the non-working app.

0 Karma

Path Finder

Another wise response,

When I deployed the _knowledge only app I made sure to copy the metadata. As you can see they are the same in both apps:

-bash-4.2$ cat Splunk_TA_aws*/metadata/*
# Application-level permissions

[]
access = read : [ * ], write : [ admin, power ]
export = system

[views/setup]
export = none

[html/Configuration]
owner = none
export = none

[html/Inputs]
owner = none
export = none

# Application-level permissions

[]
access = read : [ * ], write : [ admin, power ]
export = system

[views/setup]
export = none

[html/Configuration]
owner = none
export = none

[html/Inputs]
owner = none
export = none
0 Karma

Esteemed Legend

You are killin' me.

0 Karma

Esteemed Legend

Are you using Splunk Enterprise Security and not the latest version and are you searching inside of that app scope? If so then you need to modify the whitelist to allow your app to work inside of ES:
https://docs.splunk.com/Documentation/ES/4.0.1/Install/InstallTechnologyAdd-ons#Import_add-ons_with_...
In the newest version of ES, this feature has been removed.

0 Karma

Path Finder

An interesting perspective but no, ES is not on either of the search heads I am using to test.

0 Karma

Esteemed Legend

The app name definitely does not matter because the original is in $SPLUNK_HOME/etc/apps/whateverapp/default/whatever.conf (default) and the replacement is in $SPLUNK_HOME/etc/apps/whateverapp/local/whatever.conf (local) so the whateverapp makes no difference at all. My suspicion is that you are not deploying the apps to the correct place. If these are index-time extractions, then they must be deployed to the first full instance of splunk that handles the event (usually HF-tier or IDX-tier). If these are search-time extractoins, then they must be deployed to the Search Heads.

0 Karma

Path Finder

Hi,

Thanks for your response. I am certain that these are search-time extractions. As I said above, if I simply move the config from one app to the other on a single search head and restart everything works on that search head.

0 Karma