Security
Highlighted

Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Builder

Hi,

On Splunk Cloud, the admin role has by default access to all non-internal indexes. At a customer's site, we want to retain the the access rights of admins by keeping "available indexes" (with the exception of internal indexes) in their own "access roles." Thus, our admin role would only grant the user the capabilities of the admin, but not any non-internal indexes. Those indexes would have to be granted by their own 1:1 access roles.

Our problem is that when we grant a user the custom admin role we've created, that user can't see the graphs in the "distributed management console." It seems like the users aren't allowed to see the _internal index, even though it clearly says in the custom admin role we've created that all internal indexes should be searchable. We've even added the index _internal specifically, in addition to "all internal indexes," just to be sure. What's interesting is that if we let our custom admin role inherit from the default admin role, then it works.

Any idea of what could be the problem here? Could there be some sort of "hidden" access right or capability on the default admin role in Splunk Cloud?

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Splunk Employee
Splunk Employee

It sounds like there is some difference(s) between the out of the box admin role and the custom admin role. Here's my suggestions to review/change and these are all under Settings->Access Controls->(Custom Admin Role):

1) Under -> Indexes Searched by Default
- Make sure "All Internal Indexes" is selected

2) Under -> Indexes
- Make sure "All Internal Indexes" is selected

If these do not resolve your issue, check the list of capabilities between the original admin role and your custom admin role - if any capabilities are missing when comparing against the original admin role, add those missing capabilities to the custom admin role.

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Builder

Hi. Thanks for your answer! I was thinking the same thing, there must be a difference between the two admin roles. I've triple checked that all capabilities from the default admin are copied over to my custom admin, also regarding inheritance from user and power roles. The custom admin have access to "all internal indexes," and "all internal indexes" are searched by default.

To me it almost seem like other roles than the default admin role on Splunk Cloud aren't allowed to search on internal indexes, even though the role specifies that they should be able to do so. Please, prove me wrong. I'd like to get to the bottom of this.

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Splunk Employee
Splunk Employee

So, currently a user under the custom admin role is unable to see the graphs in the DMC - but if they execute a search of "index=_*", do they see results?

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Builder

No, my custom admin role don't see any internal data at all, e.g. index=_internal.

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Splunk Employee
Splunk Employee

Are you receiving "No results found" after the search, or nothing at all? Can that user search any data/index at all?

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Builder

I receive "no results found" when searching for internal indexes. Searching for non-internal indexes works just fine.

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Splunk Employee
Splunk Employee

Ok, thanks for your prompt response on that...

So, I tested your scenario out in a Splunk Cloud instance - creating a custom admin role, assigning no inheritance, assigning specific capabilities (1-for-1 match of default admin role capabilities), and selecting "All internal indexes" under "Indexes searched by default" and "Indexes" sections. Then, I created a new user and assigned that user the custom_admin role.

After saving it, I was able to login as the user assigned to the customadmin role and successfully receive results from a search of "index=*".

I'm posting my authorize.conf below (kinda lengthy) - compare this list to your authorize.conf, add any capabilities missing from your custom admin role to your role. You should be able to find yours under $SPLUNK_DB/etc/system/local/authorize.conf.

Also, if this doesn't work for you, you may want to at least setup inheritance from the "user" or "power" role.

[rolecustomadmin]
acceleratedatamodel = enabled
accelerate
search = enabled
adminallobjects = enabled
canownnotableevents = enabled
change
authentication = enabled
createexternalticket = enabled
cumulativeRTSrchJobsQuota = 400
cumulativeSrchJobsQuota = 200
editcorrelationsearches = enabled
edit
deploymentclient = enabled
edit
deploymentserver = enabled
edit
distpeer = enabled
edit
forwarders = enabled
edithttpauths = enabled
edit
inputdefaults = enabled
edit
logreviewsettings = enabled
editmodinputthreatlist = enabled
editmodinputwebping = enabled
edit
monitor = enabled
editnotableevents = enabled
editperpanelfilters = enabled
edit
postprocess = enabled
editreviewstatuses = enabled
edit
roles = enabled
editscripted = enabled
edit
searchheadclustering = enabled
editsearchscheduler = enabled
editsearchserver = enabled
editserver = enabled
edit
sourcetypes = enabled
editsplunktcp = enabled
edit
splunktcpssl = enabled
edit
suppressions = enabled
edittcp = enabled
edit
tokenhttp = enabled
edit
udp = enabled
edituser = enabled
edit
viewhtml = enabled
edit
websettings = enabled
embed
report = enabled
getdiag = enabled
get
metadata = enabled
gettypeahead = enabled
indexes
edit = enabled
inputfile = enabled
license
edit = enabled
licensetab = enabled
list
deploymentclient = enabled
list
deploymentserver = enabled
list
forwarders = enabled
listhttpauths = enabled
list
searchheadclustering = enabled
listsearchscheduler = enabled
outputfile = enabled
pattern
detect = enabled
requestremotetok = enabled
restappsmanagement = disabled
restappsview = disabled
restpropertiesget = disabled
restpropertiesset = disabled
restartsplunkd = enabled
rtSrchJobsQuota = 100
rtsearch = enabled
run
debugcommands = enabled
schedule
search = enabled
search = enabled
srchDiskQuota = 25000
srchIndexesAllowed = ;_
srchIndexesDefault = ;_
srchJobsQuota = 50
srchMaxTime = 0
srchTimeWin = 0
web_debug = enabled

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Builder

Thank you for checking this out for me! How do you see your authorize.conf while using Splunk Cloud? As far as I know the only way to do it is to send a request to support. I will do so, and then I'll cross-check my configuration with yours. I'll get back to you.

0 Karma
Highlighted

Re: Why is our custom admin role unable to search _internal index data in Splunk Cloud?

Splunk Employee
Splunk Employee

You're correct that you don't typically have access to the authorize.conf within Splunk Cloud, and I forgot about that as I sent my previous post - sorry about that. The one I have access to is a unique situation, so typically no access is available.

You can compare what I sent you against your custom role within the UI: Settings -> Access Controls ->

Note: The srchIndexesAllowed and srchIndexesDefault in the listing above relates to the indexes sections at the bottom of that screen, in which mine has access to All Internal Indexes and All Non-Internal Indexes for both.

0 Karma