After adding a user to only one search head in a search head cluster, why is the user unable to see all saved search results?

[ishaanshekhar]

I have a search head cluster and have created a custom role (authorize.conf), which has been deployed to each SH through a custom app.

I added a user "xyz" to only one SH so that the user only uses a particular SH. Everything seems fine except that the user is not able to see all the saved search results.

Error message when I use loadjob command:

Error in 'SearchOperator:loadjob': error accessing https://127.0.0.1:8089/services/search/jobs/scheduler__admin__search__RMD5fc0cc9974bfd0925_at_145320..., statusCode=403, description=Forbidden

However, when I added the user to all the SHs, there were no errors.

My question is, did the issue happen because of not adding the user to all SHs, or because of a capability issue in authorize.conf?

Thanks
Ishaan

[renjith_nair · Accepted solution]

In a search head cluster, if you are using local splunk authentication, then the users should be created separately on each member. The user on a search head is local to that node and its same as with the objects created by the user unless the user has admin privileges.
It's always advisable to configure a central user base/authentication system like LDAP in a search head cluster.

http://docs.splunk.com/Documentation/Splunk/6.2.2/DistSearch/AdduserstotheSHC

[ishaanshekhar]

Thanks a lot, @renjith.nair

Is there any privilege that I could add to the user role that would let them see all the saved search results from remote SH, or I should avoid this path?

More importantly, we are planning to move the authentication to reverse proxy module. This is important to us so need your suggestion. Kindly let me explain...

On each SH, there will be a reverse proxy module (which will handle our SSO) and a scripted authentication on splunk side.
In short,

[USER] ---> Company specific SSO URL ---> Load Balancer ---> Reverse proxy on a SH ---> Scripted Authentication on that SH

My question is, in this setup, the users will not be manually added to any SH and we would let the script determine the role for a user (scripted authentication on splunk side using python). Will we not face the same issue again where users are not able to see savedsearch results from remote SH? Or, would the remote SH also check its local scripted authentication file in this case, so that the user is okayed?

Appreciate your help! Thanks in advance.

Thanks
Ishaan

[renjith_nair]

There is one more thing we have to consider about saved reports. On a search head cluster, the saved search can execute randomly on any member and the result will be stored on local disk. So the link to this result might not work on another member. For this you have to set embedSecret in server.conf

embedSecret = <string>
* When using report embedding, normally the generated URLs can only
  be used on the search head they were generated on
* If "embedSecret" is set, then the token in the URL will be encrypted
  with this key.  Then other search heads with the exact same setting
  can also use the same URL.
* This is needed if you want to use report embedding across multiple
  nodes on a search head pool.

[ishaanshekhar]

Thanks! @renjith.nair

I made a custom app, made it global using default.meta, and deployed to the SHC.
server.conf
[general]
embedSecret=mySecretKey

Just to let you know, we have quite a few savedsearches in our cluster and we display all the results on a single page. The suggested config change seems to be partially working for the user on the lone SH. I am getting more results than before but still getting the error message for some or the other savedsearch at random times.

Here is the error message:
Error in 'SearchOperator:loadjob': error accessing https://127.0.0.1:8089/services/search/jobs/scheduler__admin__search__RMD5f5180dd419bca903_at_145328..., statusCode=403, description=Forbidden

Did I miss something in the process or is there any config or role change required? Thanks a lot, in advance.

Thanks
Ishaan

[ishaanshekhar]

@renjith.nair Hi Renjith, could you please help? Thanks a ton.

[renjith_nair]

First of all, the embed report is needed for report embedding as mentioned before. It's not mandatory for all SHC. I have just mentioned it for your reference,
In the above scenario, you have created a user (lets say 'X') in one search head and not on others and trying to access the saved search created by X on all search heads. Is that right? From the error message,it looks like a permission problem. Try accessing all the search resulst with an admin user and see if it works. If it's working , then share the saved search with the user you are trying to access the results.

[ishaanshekhar]

Hi @renjith.nair, thanks for your response.

A few points:

1. All saved-searches are owned by admin level user with read permission to all.

2. Requirement: A non-admin user "X", who is added to only one SH needs to read the saved-search results (no matter the saved-search runs on local or remote SH).

3. Currently, non-admin user "X" is not able to see all the saved-search results. However, another non-admin user "Y" is able to see all the results. Both "X" and "Y" share the same role privileges. The only difference is that "X" is added on only one SH, whereas, "Y" is added on al SHs.

   1. Additionally, an admin level user "A" is able to see all results, even if "A" is added to only one SH.

That is why I am wondering how to make "X" see all the results even though it is not added to all SHs. May be add any particular privilege to the user role or any other setting?

Thanks a ton,
Ishaan

[splunk_zen]

I'd say it's the embed_report capability which is allows the Admin level users to see those search artifacts.
As renjith mentioned, it really doesn't make sense you having disparate user lists across SH cluster members and should rather use LDAP to manage which users have access to what.
You'll keep running into different issues until you move away from local users management.

[ishaanshekhar]

Thanks a lot, @splunk_zen and

@renjith_nair ... sorry, I was trying to delete my old comment and that deleted your comment in the below chain, but what you said was correct. I am not seeing any issue while using scripted authentication now.
Earlier, it might be because not all SHs were not scripted authentication. I made all of them consistent and I have not faced such issue till now.

[ishaanshekhar]

Hi @renjith.nair ! Please help!

Thanks
Ishaan