topic How to route to index by Source in Getting Data In

How to route to index by Source

danielrichards — Thu, 09 Dec 2021 10:44:18 GMT

Hi All,

Having an issue trying to route events to an index by source, posting as a new question as I've not found anything that's helped me understand how /where to configure this.

We have events being streamed to HEC (Token) hosted on a HF, which is then forwarding the events to an Indexer, all events are ending up in the Main index on the Indexer.

How can events of the default field Source 'xyz' be sent to a specific Indexer Index 'index_xyz'?

I've seen numerous posts about routing to a specific Index using the SourceType but not Source. I know props.conf and transforms.conf are needed but I've not seen any examples for using Source, also I'm unsure whether they should be implemented on the HF or the Indexer...

The resoning for using Source for routing to a specific index is that these events are always lsted as the Token Name 'xyz'.

TIA

Daniel

Re: How to route to index by Source

gcusello — Thu, 09 Dec 2021 10:57:03 GMT

Hi @danielrichards,

the approach to indexes choice should be: to put in the same index events with the same retention and the same access rules.

It isn't so relevant but it isn't i a good idea to put different sources in different indexes because you'll have to manage more indexes than required.

If you want to do this, you can override the index in two ways:

putting the index value in the HEC inputs,
overriding the value in the indexers (or Heavy Forwarders when present) following the instructions at https://community.splunk.com/t5/Getting-Data-In/How-can-I-override-an-index-name-based-on-sourcetype/td-p/161444 .

Ciao.

Giuseppe

Re: How to route to index by Source

isoutamo — Thu, 09 Dec 2021 11:37:40 GMT

And use source instead of sourcetype is shown here: https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

With HEC if you have several tokens based on source then the easiest way is set it in inputs.conf as @gcusello said.

r. Ismo

Re: How to route to index by Source

danielrichards — Thu, 09 Dec 2021 11:50:57 GMT

Hi Giuseppe,

I had tried to specify the index in the HEC Inputs without success. Not sure why.

Following the instructions at https://community.splunk.com/t5/Getting-Data-In/How-can-I-override-an-index-name-based-on-sourcetype... . worked, thanks.

Interestingly it only works when defining the props.conf & transform.conf on the HF, and not the Indexer...

Re: How to route to index by Source

gcusello — Thu, 09 Dec 2021 12:01:27 GMT

Hi @danielrichards,

you have to make the overriding on the first Splunk full system (not UFs) that cooks the ingested logs.

In other words: if you have an HF you have to put conf files on it, except if you don't send cooked logs, in this case you have to put them on Indexers.

I usually put these conf files both on HFs and Indexers to be more sure.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors. 😉