We have recently cleaned up our Splunk eco-system and developing process / procedures for our we manage it. I wanted to get some ideas of questions Splunk Admins ask teams during the data on-boarding phase. Questions that make sure teams know their data and will get value out of the tool as opposed to using Splunk as a dumping ground.
Here are some of the questions I use:
These are the only critical ones I have come up with so far. I would love to hear what you all ask!
One of the most important questions I've learned to ask users who are asking for data onboarding:
"Do you have a service architecture diagram that you can share with me?"
Knowing where the system(s) are in relation to the service, and how data is passed around, is more than half the battle in building a complete service/ops/security package.
Other questions I like for all data sources:
"Where are the logs located on your system?"
"What are the log files you want monitored, and what are their contents?"
Asking these questions is equal to having a sample of the log, because it tells you what the end user thinks is going on, and you can look at reality to see if they are right, close. or have no clue. This is important to know if you are going to be building reports and dashboards for them.
There is a wide range of states between a dumping ground and a perfect Splunk implementation and I think most of us live somewhere in between. The "common" challenge, in my mind, is to move from the dumping ground scenario to the perfect Splunk implementation.
Here is a sample form that I love to start with with. (Shout out to @LTRand for hooking me up with this list initially - see his github below for another take on an onboarding form)
This is not an all encompassing list, but should get you moving in the right direction.
Implementing this into a self serve form in a portal is a great idea as well! (ie. SNOW, Remedy, whatever you use for workorders)
Allowing the groups to complete the form when it suits them, should trigger a fulfillment request or ticket and a quick working session with the requestor once the admin has had a chance to review.
Having a getting started package that links to any internal chat rooms/community, docs and training you have, or to splunk fundamentals 1 free training, and most importantly, a use case showcase showing how teams are extracting value is a great idea as well!
The main thing is to stress that onboarding is the just the beginning of the process. Once the data is in, the real fun can start!
Name/Owner: __________________________________________________ Title/Role: __________________________________________________ Team: __________________________________________________ A data sample. Description of the data: Sourcetype suggestion: _________________________ How are events broken? ___ single-line ___ multi-line (events start with: _________________________) Is there a date/timestamp? ___ yes ___ no ___ >1 (pick one: _________________________) What time zone is in use? _________________________ What fields are interesting? ______________________________________________________________________ Uses for the data: Searches ___ I want to search using keywords for troubleshooting ___ I want real-time searches ___ I want to compute statistics over the last _____ mins/hours/days/months ___ I want to know the top n of something over the last _____ mins/hours/days/months ___ I want to create and save my own searches Reports ___ I want to create charts/tables/gauges over the last _____ mins/hours/days/months ___ I want real-time reports ___ Please give me a dashboard ___ I want to create and save my own reports ___ I am building reports over long periods of time and want data summarized Alerts ___ I want Splunk to send me alerts via email every _____ mins/hours/days Clues on data collection: Where is it located? server(s) _________________________ path _________________________ How should it be collected? ___ Splunk Universal Forwarder ___ syslog ___ other: _______________ Hints on retention policy: Keep it for this long: _____ days/months/years Store this much of it: _____ MB/GB/TB Who should have access to the data: Team / LDAP Group: ____________________ Apply the Common Information Model: Is there a TA available (look on Splunk Apps)? Validate success of data on-boarding