topic Re: Index override on heavy forwarder data in Getting Data In

Index override on heavy forwarder data

yu94 — Fri, 20 Oct 2017 11:03:45 GMT

Hi,

There is situation where we have installed DB connect on HF and then the HF sends that data to 2 sets of different indexers and now we need to override the index name at one set of indexers .

We have tried to override the index which is coming from UF is working fine.

When we tried to override the index which is coming from HF (DB Connect), it was not working may be due to the metadata already set by this HF.

Can you help me to fix this issue?

Regards,
Thippesh

Re: Index override on heavy forwarder data

gcusello — Fri, 20 Oct 2017 11:12:59 GMT

Hi yu94,
the easiest way is to override the index name in the set of indexers:
on transforms.conf

[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index

on props.conf

[mysourcetype]
TRANSFORMS-index = overrideindex

Bye.
Giuseppe

Re: Index override on heavy forwarder data

yu94 — Fri, 20 Oct 2017 11:20:24 GMT

Hi Giuseppe,

We have tried this. This is not working because the data which generates is HF and we tried to override at the indexer, so on the HF it is passed the parsing queue and set the metadata file so will not be able to override at the indexer.

Any other ways of doing it?

Regards,
Thippesh

Re: Index override on heavy forwarder data

gjanders — Fri, 20 Oct 2017 11:36:30 GMT

Since this is a heavy forwarder the data is cooked on this server, therefore the index setting is done here.

Is it possible to get both sets of indexers to use the same index for this data?
If not the transforms.conf might work but you would need to do the transform based on which output was chosen, and I'm unsure how to do that part, also DB Connect can set an index, but it will only set one per input.

Perhaps you can change your indexers to have the same index name?

Re: Index override on heavy forwarder data

gcusello — Fri, 20 Oct 2017 11:41:38 GMT

Hi yu94,
Did you tried to override index on indexers? you said that you want to override index only on a set of indexers, so you can:

take logs using DB Connect on HF setting index1,
then on indexers (only the ones of the selected set), use the index override configuration.

In this way you can ingest logs on HF with the original index and then modify it on the indexers.

Bye.
Giuseppe