Hey Splunk developers,
Quick question for you guys, I'm curious if it would be considered against best practices, or an anti-pattern to add your own code to the props.conf / transforms.conf file of a Splunk supported technical addon in order to add support for your own custom sourcetypes? Specifically, The Splunk Addon for Microsoft Cloud Services.
My use-case is this, I'm utilizing a "custom" event delivery mechanism to send Azure Diagnostic Logs generated by my resources in my Azure tenant to Splunk over HTTP Event Collection (HEC). The mechanism consists of an Azure Event Hub that receives diagnostic events from my resources in Azure. Downstream an Azure Function App monitors the upstream event-hub namespace for incoming events, parses those events, constructs a source-type based on the data in the parsed event, and then wraps the event payload in a HTTP request and sends it to Splunk. The code for the Azure Function App can be found here.
The general source-typing logic for the function application is below.
Functions that collect diagnostic log data attempt to construct a sourcetype based on the resourceId of the event. The logic for this sourcetype construction can be found in the getSourceType function in the ./helpers/splunk.js file. The following steps are used to construct the sourcetype: A regular expression is used to extract two groups after the text /PROVIDERS Example /PROVIDERS/MICROSOFT.RESOURCES/DEPLOYMENTS/ Periods (.) and forward slashes (/) are replaced with colons (:) The event category is appended Example An event with a resourceId of /SUBSCRIPTIONS/subscription ID/RESOURCEGROUPS/group/PROVIDERS/MICROSOFT.RESOURCES/DEPLOYMENTS/FAILURE-ANOMALIES-ALERT-RULE-DEPLOYMENT-12345678 will have a sourcetype of azure:resources:deployments:administrative If a sourcetype cannot be constructed from the event, the specified default sourcetype entered at setup will be used.
Here's my dilemma, the team that administers our corporate Splunk instance would like to utilize as many TA's as possible to ensure CIM compliance of on-boarded data with as little work as possible, completely understandable. Therefore, for certain event sources I've hard-coded the source-type logic in my function app to match several of the pre-defined source-types found in the TA I linked above ("azure:monitor:aad" & "mscs:azure:security:alert" as examples).
However, we're at the point where I'm trying to on-board data that does not have a pre-defined sourcetype within the TA's we're utilizing. As an example, Azure Front Door Firewall Logs which will sourcetype to "azure:network:frontdoors:frontdoorwebapplicationfirewalllog".
To work around this, I've suggested to our content development team that we begin adding logic to the props.conf and transforms.conf of the TA listed above that will match the sourcetypes my function-app is sending over to Splunk and I've essentially been told no, as this isn't best practice.
Is my suggestion truly against best practice, or an anti-pattern? In my mind the only extra overhead to take into consideration is merging our custom props.conf / transforms.conf with the base TA whenever an update for the TA is released.
What's my best path forward here?
As with any such questions - it's always a matter of weighing all pros and cons. There are no simple answers since _in some cases_ _for some customers_ some practices even though generally not advised could be the proper way to go. So it's hard to give a generic yes/no answer in such cases.
Having said that - I'm generally against modifying official add-ons if you can avoid. If you do modify the add-on, it's all yours in terms of maintenance. So every update _you_ have to make sure nothing is broken.
That's one thing. So if you add new sourcetypes, the natural approach would indeed be to create a new TA.
On the other hand, if they are used with some custom extensions of a solution for which there is already a well-known TA it can be tempting to add them to the existing TA because of "consistency". But carefuly consider the maintainability of such solution.
Another thing - I saw "hardcoding" written in your question. Remember that abstracting and externalizing configuration is generally a good thing so I'd say hardcode as little as possible. Use datamodels, eventtypes, tags, aliases. Make your creations flexible and reusable.
IMO, a new sourcetype should be a new TA. Howver, it's reasonable to add to an existing TA if that TA is already doing work that would be hard to replicate - such as collecting data using an API.
If you choose to add your props and transforms to an existing add-on, do so in the local directory of the add-on. That will prevent overwriting of your changes when the TA is updated and avoid the need to merge changes.