Data Onboarding Strategy

[pradeepkumarg]

I would like to hear from other admins on how they are keeping up with high demand of data onboarding requests into their Splunk instance in large organizations.

We are battling with more than 300 requests per month to onboard data into Splunk as every application in the organization wants to utilize Splunk for monitoring and the demand only keeps increasing. Most of these are custom application logs. The biggest bottleneck is  defining props (LINE BREAKER, TIME STAMP etc..,) for the source types by having to manually analyze each individual log. Other parts of the onboarding (inputs.conf, indexes.conf etc..,) can be easily automated for seamless onboarding but not props.

Not defining props for source types to leave to Splunk defaults is not an option as we have seen some serious performance issues on indexers.

I would like to hear from Splunk if there is a strategic direction in this regard to make admins life easier with respect to onboarding and other admins who might have dealt with similar situation and overcome in creative ways. 

Regards,

Pradeep

[thambisetty]

Even I have same problem like we get security logs from different applications on monthly basis.

The applications are customer specific so they don’t have proper logging mechanism. Sometimes we had to get them from debug logs where each event would further split into many events.
what I do is I onboard all the logs required for security monitoring, identify the required event types by just simulating few scenarios where high risk is involved and apply props|transforms, the rest of events which are not matching with my event types will be discarded.

I know this is not a solution for your question. Kind of feedback you can consider.

There is no such option to automate props and transforms on unknown data/events.

we can can automate field-aliases if the event is in key-value fair, not tagging of events. tagging of events is most important to make use of DATA MODELS.

[isoutamo]

Hi

In our experience there are something what you can do and request from you "clients/customers" which want to onboard data.

1. Create and update instructions to end users and developers
2. Create integration catalog where all relevant information about onboarded systems.
3. Create common templates for that
4. Create Naming standard for all Splunk related stuff
5. Create separate TAs + Apps for common cases
   1. Usually there are some frameworks how in-house and some other apps e.g. logging stuff
   2. Try to endorse other to use these and create some guidelines for these
6. Create separate TAs + Apps for all onboarded systems and use above common as much as possible
7. If possible outsource at least inputs.conf to your clients
   1. maybe even creating props.conf + transforms.conf, BUT you must check and install those to the general IHF/HF/IDX
8. Use git or other version management system to keep track of all configurations
9. Deploy configurations as TA/Apps to splunk from git

A good place to learn more is https://lantern.splunk.com/hc/en-us/sections/360007451513-Data-Management and whole Splunk Success Framework at lantern.splunk.com.

r. Ismo

[pradeepkumarg]

@isoutamo  thanks for your response. We follow most of these guidelines if not all.  The guidelines is a generic framework and does not address my problem. We encourage our SMEs to provide props.conf during the on-boarding process but I can't enforce it.  We try to reward by fast tracking the on-boarding process for them.  There are only handful of them who provide us props upfront. But this is not sufficient to keep up with the demand. I am looking for a more self-service on-boarding by SMEs approach and we as admins to do governance on the data and keep a check on the license and hardware.