topic Re: How to separate data in Splunk and what documentation covers this information at the forwarder and indexer level? in Getting Data In

How to separate data in Splunk and what documentation covers this information at the forwarder and indexer level?

lacat — Wed, 21 Jan 2015 18:13:54 GMT

Hello, I'm relatively new to Splunk. The company I work for is in the process of deciding between using Splunk or ELK.

Our requirements are to collect data from many machines/servers and strictly separate data by e.g. department or branch office.

In my opinion the Splunk architecture looks like this:
A client has a forwarder installed on it. This forwarder sends data to a specific heavy forwarder.
The HF then parses data to an indexer where it is indexed and later can be searched by the search application.

I didn't find anything about how to separate data in the Splunk documentation.
Does anyone know where you could start the separation?
Is it possible to start it right at the HF?
Does anyone have a link to documentation for it?

Thanks in advance!

Re: How to separate data in Splunk and what documentation covers this information at the forwarder and indexer level?

chanfoli — Wed, 21 Jan 2015 18:30:56 GMT

Typically I see the type of separation you are talking about being done at the index level. I.E. you set up indexes to apply access permissions, retention policies, etc on the different classes of data you are indexing. Heavy forwarders can route data to departmental indexers if you need physical separation. If you just need to control access, this can be done at the role level using shared hardware and using appropriately named and designed indexes. Heavy forwarders are not necessarily a requirement unless you want physical separation and you have inputs from source devices on which you can't install forwarders. The right documentation depends on the chosen approach.

Re: How to separate data in Splunk and what documentation covers this information at the forwarder and indexer level?

chanfoli — Wed, 21 Jan 2015 18:35:57 GMT

Additionally, here is the basic documentation which will get you started in understanding index-based separation: http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Setupmultipleindexes

And here is documentation regarding routing which might be your approach if you need to have a HF direct different events to different physical indexers due to policy constraints:
http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Routeandfilterdatad

Re: How to separate data in Splunk and what documentation covers this information at the forwarder and indexer level?

ppablo — Wed, 21 Jan 2015 18:37:08 GMT

The architecture you described above is just one of many possible ways of forwarding, parsing and indexing data. Here's a page from Splunk's documentation on various ways of routing and filtering data.
http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Routeandfilterdatad

Re: How to separate data in Splunk and what documentation covers this information at the forwarder and indexer level?

somesoni2 — Wed, 21 Jan 2015 18:44:55 GMT

The splunk architecture could be UF-> Indexer OR UF-> HF-> Indexer. To separate/group data by Department/Branch Office, you could assign appropriate sourcetype OR store them in separate index (this way you can setup security very easily as well).

Re: How to separate data in Splunk and what documentation covers this information at the forwarder and indexer level?

lacat — Wed, 21 Jan 2015 22:18:14 GMT

Thank you for your help 🙂