Restricting or not restricting users from creating scheduled searches?

[mfrost8]

We currently don't allow anyone but admins to create scheduled searches. I'm sort of torn on this one as I don't want this policy to be a barrier to doing cool things with Splunk. On the other hand, I don't have a lot of confidence that people won't create bad searches that, say, run against All Time and chew up resources. (We restrict real time searches for the same reasons).

Only allowing admins to create scheduled searches means that we can sanity check those searches before letting them run. Perhaps more importantly, we don't have to police people's searches when we do this.

I realize that we can use some of the built-in views to monitor which searches are taking the most time, but that means policing and tracking down offenders, etc.

But maybe this is something I'm thinking is a much bigger issue than it is.

I'm curious what other Splunk customers do. Do you let everyone created scheduled searches?

Thanks

[kmattern]

We don't let anyone but admins create scheduled searches. In fact we have locked Splunk down where they can't even get a search bar. We have almost 40 apps running and have removed anything that a user can click on, such as "view results" as well as all menu options that we have not created, including manager, jobs, logout, etc. We’ve also removed the Save and Create buttons from all forms. In fact we have our own logout link, dashboard banners, login banner etc. We have even edited "No results found" to "Nothing to report". Of course we are doing this for our DoD customer and have to insure that the Navy doesn't see Army dashboards and that Army users can see only dashboards relating to their assigned weapon system. Next on our list is implementing SSO.

As a result we have done a lot of customization, even down to the Python level, and will do more. I have written a Windows program that I use all the time to search various parts of Splunk, including my own apps, for specific keywords by file type. Very handy program, I can’t live without it.

[kmattern]

Actually we're looking to put as much output into outputlookup tables CSV) as we possibly can. Then we'll write our own asp or java scripts to generate output from those lookups so we can embed specific tables into our portal pages and bypass the Splunk interface whenever possible. Then we'll schedule the searches to run and gather the results. I don't think we have begun to make a dent in the customization we need to do. Mostly because of security concerns.

[mfrost8]

Thanks for the feedback, kmattern. I'd say we're definitely somewhere between freewheelin' for scheduled searches and the lockdown you need to implement. It's always interesting to see how other sites set up their deployment.

[carmackd · Accepted solution]

I know this is a bit late, but I found this post interesting and felt I could add something to the topic by sharing how we do things.

We removed the search scheduling capability from the default user role and created a new role called "schedule_search". If needed, a user can request the schedule_search role, but is only granted if the user has previously taken our in-house Splunk training course. Like the posters method, we also allow users to submit their searches for review and we (admin's) schedule if deemed efficient enough, or will tweak until so. We also have a few daily scheduled searches to monitor any user schedule search that runs too often or for too long. If they should show up on this report, we investigate and make the appropriate changes, if needed.

This method works pretty well for us.

Thx

[mfrost8]

carmackd,
Roughly how many users do you have, if you don't mind my asking. Thanks.

[mfrost8]

Thanks. I saw this as something that seemed like a necessity (i.e. keeping just anyone from scheduling searches), but wasn't sure if in the real world people were experiencing issues with this and thus having to limit scheduling of searches.

[lpolo]

You can use roles for each set of users. By default you have these set of roles:

admin -- this role has the most capabilities assigned to it.
power -- this role can edit all shared objects (saved searches, etc) and alerts, tag events, and other similar tasks.
user -- this role can create and edit its own saved s

More information:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Addandeditroles
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Aboutusersandroles

[mfrost8]

Thanks. I'm less concerned about access to objects than I am about poorly designed searches that chew up resources. I wouldn't want to find 4 simultaneous "All Time" searches running on my search heads the week I let people make their own scheduled searches.

Again, just looking for what other Splunk admins do with their users in the real world.

[lpolo]

I would recommend to normalize all the splunk instances to clearly identify all host, source logs, and sourcetypes.
In this way, you have the capability to know what type of information each set of users can access. With this information in place, you could define the Splunk object access rights for each set of users by using roles.

Explore Access Control Matrix to document the rights for every set of users.

[mfrost8]

Thanks. My question isn't so much about how to let people make scheduled searches. I know how all that works. My question is how people really do this in the real world. It seems iffy to me to open this to anyone and also trying to figure out who will be good and who wouldn't also seems like a challenge. I would say this is more of a policy question than a "how can this be done in Splunk?" question.

What do customers do in this regard in the real world?