topic Re: Optimising redirection of an index in Getting Data In

Optimising redirection of an index

m91886 — Wed, 30 Sep 2020 00:07:20 GMT

I am redirecting an index however, I would like to possibly increase performance.

My props.conf looks like this:

[host::MM[0-9]{6}-PC]
TRANSFORMS-index = overrideIndexoldIndex

transforms.conf looks like this:

[overrideIndexoldIndex]
DEST_KEY =_MetaData:Index
REGEX = oldIndex
SOURCE_KEY=_MetaData:Index
FORMAT = newIndex

My understanding is that it is applying this transform for all data from host:MM[0-9]{6}-PC. The transform is just redirecting index:oldIndex to newIndex. There is a lot of data from hosts that matches this criteria. Is there a way to first check that the index is oldIndex and than look for those hosts and apply the transform then. Logically this would increase performance as there is far less data being sent to the index oldIndex than, the data being sent from those hosts that match our criteria.

Essentially I would like to understand the parsing of data better surrounding transforms and if this is a valid optimization how to go about implementing it.

Re: Optimising redirection of an index

somesoni2 — Thu, 18 Apr 2019 19:26:32 GMT

You current configuration (assuming it's been placed on the instance that does the parsing i.e. heavy forwarder or indexer whichever comes first) override index name to newIndex for each event tagged with index=oldIndex and coming from hosts matching pattern MM[0-9]{6}-PC. Unfortunately, this override can only be setup at sourcetype, source OR host level, and not at index level.

Any specific reasons for overriding index for those host/index combination? Could you explain your requirement little more in detail?

Re: Optimising redirection of an index

m91886 — Fri, 19 Apr 2019 01:56:19 GMT

Actually, the sourcetype would work. The reason is that a group is sending splunk logs to two environments. On one those environment's logs are set for oldIndex and on the other environment we want them in newIndex. Since the universal forwarder only sends those logs using one index to both environments we are using the transforms to change the index in our environment.

Would this be valid?

[sourcetype:OriginalSourceType]
[host::MM[0-9]{6}-PC]
TRANSFORMS-index = overrideIndexoldIndex

Re: Optimising redirection of an index

woodcock — Fri, 19 Apr 2019 13:06:23 GMT

Your logic and configurations are correct and it cannot be done any other way, other than by source or by sourcetype instead of by host in props.conf.

Re: Optimising redirection of an index

somesoni2 — Fri, 19 Apr 2019 13:33:47 GMT

For sourcetypes, you don't need the prefix. The stanza name should be like this

[source::<source>] OR [host::<host>] OR  [<sourcetype>]

Re: Optimising redirection of an index

Anam — Wed, 24 Apr 2019 16:46:24 GMT

Hi @m91886

Looks like you have a few possible solutions to your question. If one of them provided a working solution, please don't forget to click "Accept" below the best answer to resolve this post. If you still need help, please leave a comment. Don’t forget to upvote anything that was helpful too. Thanks!