In Splunk Enterprise Security, what are the pros and cons of querying across multiple Splunk systems?

[a212830]

Hi,

We have multiple Splunk systems across different business units, managed separately. Our ES Splunk has a requirement to read data that is indexed in a non-ES system. Is there any reason that we can't just add the non-ES system as a search peer and we can query the data as needed? I've done this before, but wanted to confirm it in the community.

Pros/Cons?

[johnvr]

One other possible consideration is the Asset framework in ES. Namely, are these business units sharing the same internal IP space, or is it a separate 10. or 192.168.? I also would recommend using tags for each unit, thus - as DMA summaries include tags - this would give you and others the ability to narrow down those summaries by environment.

Also, as @pkeenan87 mentioned, indexer sizing will change when you move to ES - both for the sake of adding the DMA summaries themselves onto storage, but also for throughput. A general rule of thumb is that each indexer in an ES environment should only be indexing 1/3-1/2 of what a non-ES indexer is doing, as a result of the search load.

To that end, you may find it necessary to expand the IX tier in that environment.

[johnvr]

One additional comment: @mcronkrite gave a great presentation at .conf 2016 about ES multi-tenancy. Depending on the details of how the networks are laid out, you may be looking at what's essentially a multi-tenant deployment.

https://conf.splunk.com/files/2016/recordings/enterprise-security-multi-tenant-fundamentals.mp4

[pkeenan87]

One thing to consider is the increased load the DMA searches will to the non-ES indexer and the possibility of data from that system being added to your datamodel summaries (which may or may not be desirable). This could be avoided if you have your cim_* macros tuned to only include the specific indexes that contain data relevant to that datamodel, provided there isn't an index with the same name configured on the non-ES indexer.

[a212830]

Thanks. We do a pretty good job keepin our CIM tuned to include relevant indexes, so I'm not overly concerned about that issue.

[nickhills]

No - no problems whatsoever
Assuming the hardware (ie indexers) you are querying are appropriately spec'd

[a212830]

Thanks. To setup the peering, do I need the actual admin account on the peering indexers, or is it possible to create a non-admin that has the appropriate permissions? Or any way to use keys?

[nickhills]

You do need admin credentials of the indexers to add them as peers, but it doesn't have to be 'the' admin, just a user with the admin role.

You could create a user to add them and then remove it.

[a212830]

Thanks. One last question - what about version differences? Do the systems need to be on the same rev? We are at 7.05, they are at 7.2.

[nickhills]

As a rule, SHs will complain in the message panel if the indexers are at a later revision (however in my experience, nothing bad happens, its just 'unsupported') - Not an official Splunk endorsement 🙂

Later version SH's are fine with older version indexers.

All members in either a SHC or IDX Cluster must be at the same version (including the SHC Deployer and Cluster Master)

There was a breaking change for UF->Deployment server comms around SSL (I forget the exact version numbers and the scope of the issue) but you are fine if they are all >7.x

Forwarder Version Compatibility
https://docs.splunk.com/Documentation/Forwarder/7.2.4/Forwarder/Compatibilitybetweenforwardersandind...

Splunk Component Version Compatibility
https://docs.splunk.com/Documentation/Splunk/7.2.4/DistSearch/Distsearchsystemrequirements

[a212830]

Thanks. Actually, one more question - what about different platforms? Just found out that the non-EUS platform is Windows, while ours is RHEL.

[nickhills]

Sadly, I don’t have any experience of mixed platforms with ES, but I know it’s ok with Core.

If I had to guess, I’d say it would probably work, and may even be supported, but as to performance....?

[sloshburch]

Check those pages @nickhillscpl gave for compatibility for OS as well.

If I had a nickle for every-time I told @a212830 to read the manual... he he. Hi friend!

Instructions for setting this up (like your question about credentials and stuff) should be within Deploy a distributed search environment