Deployment Architecture

Adding a Clustered Index as a Search Peer of another SHCluster

ryandg
Communicator

So this is a bit of a convoluted situation so I will try to explain as best as possible.

There are 2 Splunk environments (site1 site2) internally, both have their own set of search heads, indexers etc. Both are setup as searchhead clusters and clustered indexes.

Site1 has data that Site2 wants to be able to search, but only 2 specific indexes (indexA indexB). Site1 does not need any data from Site2.

How do we go about setting this up? Because if we add a cluster master to Site2, it gives the users access to all indexes from Site1. Even with an authorizations.conf we cannot disable Site2 from searching Site1's _internal indexes and any indexes that we share names.

The idea I currently have is adding each of Site1's indexers as a searchpeer to the Site2's SearchHead cluster and adding a user/role 'Site2User' so that when the search server is added to every indexer it looks like this:

splunk add search-server -host 10.10.10.10:8089 -auth admin:password -remoteUsername Site2User -remotePassword Site2UserPass

This would allow us to block index usage to IndexA and IndexB only and although when our indexer tier expands we will have to add more search peers it's the easiest and best method I can think of at the moment. However, does this cause issues with Site1's indexer cluster? Are there reasons we shouldn't do this? Are there better methods to achieve this?

0 Karma
1 Solution

dwaddle
SplunkTrust
SplunkTrust

Martin is right. There are two (maybe three) fundamental facts here:

[1] The process of authorizing a search head to search-peer with an indexer requires a user name and password on the remote indexer that is ONLY used to authenticate the linking process. Once the peering has been done, the account can be deleted entirely as the indexer will have a cryptographic key that can be used to authenticate the remote search head itself.

[2] From a user authentication and roles perspective, the search head has full and total autonomy. Once it is authorized to peer with an indexer, users on that search head can do ANYTHING they want to the indexers from a search perspective without the indexer's consent. By agreeing to let a search head peer with your indexer, you are confirming your total and absolute unwavering trust in the admins of that search head. There is simply no existing way for you to allow a search head to peer with your indexer and restrict the activity of its users from the indexer.

[3] When you have a cluster, pointing a search head at the individual cluster peers is not a good idea. Always configure the search head as a mode=searchhead member of the cluster and point it at the cluster master.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

Martin is right. There are two (maybe three) fundamental facts here:

[1] The process of authorizing a search head to search-peer with an indexer requires a user name and password on the remote indexer that is ONLY used to authenticate the linking process. Once the peering has been done, the account can be deleted entirely as the indexer will have a cryptographic key that can be used to authenticate the remote search head itself.

[2] From a user authentication and roles perspective, the search head has full and total autonomy. Once it is authorized to peer with an indexer, users on that search head can do ANYTHING they want to the indexers from a search perspective without the indexer's consent. By agreeing to let a search head peer with your indexer, you are confirming your total and absolute unwavering trust in the admins of that search head. There is simply no existing way for you to allow a search head to peer with your indexer and restrict the activity of its users from the indexer.

[3] When you have a cluster, pointing a search head at the individual cluster peers is not a good idea. Always configure the search head as a mode=searchhead member of the cluster and point it at the cluster master.

ryandg
Communicator

Thank you for the detailed answer. It's unfortunate because we need a way to ensure that accidents on our indexes do not occur from their team yet they need access to our data. But thank you!

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

I don't think your approach of using a low-privilege remote user is going to work. Firstly, that's only for establishing the first connection - after that the privileges (and index visibility) from the search head will be followed by the search peers. Secondly, the docs at http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/Configuredistributedsearch say you have to connect using a remote admin.

ryandg
Communicator

So then is there no way to control from the Cluster Master side what a connecting search head cluster is able to search on the indexers connected to the Cluster Master?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

As far as I'm aware, no.

Raghav2384
Motivator

How about you set the siteB's indexers in distsearch.conf? That way, unless you specify the index in the search, it won't search from the search peers of SiteB?

I have done similar in the past where two instances needed to talk to each other but then, i didn't wanted certain peers to be searched by default (Unless otherwise specified in the search by calling splunk_server="X OR y OR z")

Check out distsearch.conf http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf for more info.

Hope this helps!
Thanks,
Raghav

0 Karma

ryandg
Communicator

Biggest issue is that my team does not control SiteB so giving them the authority to be able to search our indexes needs to be limited on the connection so that there's no way they can even search our other indexes (we have sensitive data in ours that employees not within our department should be able to see).

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...