Hi Team,
We are experiencing frequent high CPU usage on Indexers and it seems like the huge factor of it are from searches with "All time" Time filter and real time searches.
With this, do we have some steps on how to restrict user on using "All time" Time filter and real time searches? Is it related to Splunk roles? and if yes, what capabilities should be remove from them so that they will not be able to use "All time" Time filter and real time searches.
Here is a very detailed answer, which will tell exactly why Real time searches suck, what do they do to your environment, why should you removed them instantly and how to remove them. Special thanks to @woodcock for this amazing answer.
Now, to get rid of All time, here's what you need to do. In the directory $SPLUNK_HOME/etc/system/local/times.conf
[other] disabled = true
. This will remove the "All Time" for all the users, including yourself. If you want to do this for particular users only, please put the above configurations under $SPLUNK_HOME/etc/users/user_name/local/times.conf. You'll have to do it for every user individually. If it's for a set of users, then please select an app, make that app the default app for all of those users and implemented the above change under $SPLUNK_HOME/etc/apps/selected_app_name/local/times.conf.
This would still allow them to use All time, if they use earliest and latest in their searches. To stop that you could do the following change under Authorize.conf
srchTimeWin =<set a value in seconds.This is the earliest time that the users belonging to this role would be able to search any data>
I'd suggest to restrict Others for all users, and leverage earliest and latest from the search yourself. Would save you a lot of time and effort in the future as well. I've done the same. Please choose accordingly.
Let me know if it helps.
Thanks,
S
** If this helps. Please mark this as an accepted answers, as it helps the future readers to find answers quickly. **
You can also limit how far back a role can search within authorize.conf (srchTimeWin). We currently do this with an app on the deployer and then push to the SHC members.
Example:
authorize.conf
[role_<role_name>]
importRoles = user
srchDiskQuota = 100
srchFilter =
srchIndexesAllowed = main
srchIndexesDefault = main
srchMaxTime = 1800
srchTimeWin = 10368000
srchTimeWin description:
srchTimeWin = <integer> * Maximum time range, in seconds, of a search. * The Splunk software applies this search time range limit backwards from the latest time specified for a search. * If a user has multiple roles with distinct search time range limits, or has roles that inherit from roles with distinct search time range limits, the Splunk software applies the least restrictive search time range limits to the role. * For example, if user X has role A (srchTimeWin = 30s), role B (srchTimeWin = 60s), and role C (srchTimeWin = 3600s), user X gets a maximum search time range of 1 hour. * When set to '-1', the role does not have a search time range limit. This value can be overidden by the maximum search time range value of an inherited role. * When set to '0' (infinite), the role does not have a search time range limit. This value cannot be overidden by the maximum search time range value of an inherited role. * This setting does not apply to real-time searches. * Default: -1
Reference:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/authorizeconf
To deny real-time searches, remove the "rtsearch" and "schedule_rtsearch" capabilities from the offending roles.
There is no equivalent for "All Time" searches. The way to avoid that is to remove it from the selector. Do that at Settings->User interface->Time ranges. Also, go to Settings->Server settings->Search preferences and make sure the default time picker is something other than "All Time".