Optimising redirection of an index

[m91886]

I am redirecting an index however, I would like to possibly increase performance.

My props.conf looks like this:

[host::MM[0-9]{6}-PC]
TRANSFORMS-index = overrideIndexoldIndex

transforms.conf looks like this:

[overrideIndexoldIndex]
DEST_KEY =_MetaData:Index
REGEX = oldIndex
SOURCE_KEY=_MetaData:Index
FORMAT = newIndex

My understanding is that it is applying this transform for all data from host:MM[0-9]{6}-PC. The transform is just redirecting index:oldIndex to newIndex. There is a lot of data from hosts that matches this criteria. Is there a way to first check that the index is oldIndex and than look for those hosts and apply the transform then. Logically this would increase performance as there is far less data being sent to the index oldIndex than, the data being sent from those hosts that match our criteria.

Essentially I would like to understand the parsing of data better surrounding transforms and if this is a valid optimization how to go about implementing it.

[Anam]

Hi @m91886

Looks like you have a few possible solutions to your question. If one of them provided a working solution, please don't forget to click "Accept" below the best answer to resolve this post. If you still need help, please leave a comment. Don’t forget to upvote anything that was helpful too. Thanks!

[somesoni2]

For sourcetypes, you don't need the prefix. The stanza name should be like this

[source::<source>] OR [host::<host>] OR  [<sourcetype>] 

[woodcock]

Your logic and configurations are correct and it cannot be done any other way, other than by source or by sourcetype instead of by host in props.conf.

[somesoni2]

You current configuration (assuming it's been placed on the instance that does the parsing i.e. heavy forwarder or indexer whichever comes first) override index name to newIndex for each event tagged with index=oldIndex and coming from hosts matching pattern MM[0-9]{6}-PC. Unfortunately, this override can only be setup at sourcetype, source OR host level, and not at index level.

Any specific reasons for overriding index for those host/index combination? Could you explain your requirement little more in detail?

[m91886]

Actually, the sourcetype would work. The reason is that a group is sending splunk logs to two environments. On one those environment's logs are set for oldIndex and on the other environment we want them in newIndex. Since the universal forwarder only sends those logs using one index to both environments we are using the transforms to change the index in our environment.

Would this be valid?

[sourcetype:OriginalSourceType]
[host::MM[0-9]{6}-PC]
TRANSFORMS-index = overrideIndexoldIndex