We've got pretty strict access rules. We have an index for each in house developed application, one index for staging, and one for production data. (approx 100 indexes). We have an index for each infrastructure "type" (cisco, aruba, paloalto, f5, checkpoint, etc.). We planned it this way because we were not sure how we wanted to handle accesses to data in the future, and since that's determined on a per index basis, we split it all out per index.
For larger user groups (Windows Server Admins, Linux Server Admins, Network Engineers), we have generalized Active Directory groups made for them. Think: "EnterpriseWindowsAdmins", "EnterpriseNetworkEngineering", etc. Users have their managers submit an access request (ticket) to our Access Management team (as an enterprise, we require manager approval for access to AD groups). We then put in place secondary approval by the Splunk Administrators. So we review and approve/deny all access requests to Splunk data for validity.
These AD groups are then mapped to a Splunk role which provides access to indexes for devices that group happens to manage. For example, the "EnterpriseNetworkEngineering" team will get access to the following indexes:
cisco
f5
checkpoint
paloalto
aruba
But they would not get access to:
windows
internal app-specific indexes
ids/ips indexes
The thought process is that they have a work-related need to access the data for the devices they manage (and technically they can access it already with admin access to these devices, Splunk is just now centralizing their data). They do not have a work-related reason to access device logs that they do not manage.
Occasionally, there is the one-off where someone from the Windows Server team may be assisting in troubleshooting a network device, or has a legitimate need that we validate to view data for that device. For this, we'll create an index-specific AD group (ex: SplunkF5Index)and process it the same way (manager approval, submits the ticket, secondary Splunk admin approval). We map that to a role in Splunk which gets access to only that index.
No users are granted Power capabilities at this time, and the Splunk team handles all "help" requests like dashboard/report creation, app installations and general help with SPL. We also are the only team that onboards data into Splunk.
For reference, we're a 350GB/day shop with about 1100 forwarders and an 8 year retention period. We've had Splunk in house for one year. We don't have a lot of technical teams that need access to Splunk, so the administration from the group/role side is pretty minimal. I approved two accesses this month to an existing group for new hires to that team. We don't provide any training and link them to the Splunk basic virtual training and most users can take it from there. Occasionally I'll look through the search history and if there's something outlandish I may shoot them an email to help them make their searching more efficient.
... View more