Getting Data In

Optimising redirection of an index

m91886
New Member

I am redirecting an index however, I would like to possibly increase performance.

My props.conf looks like this:

[host::MM[0-9]{6}-PC]
TRANSFORMS-index = overrideIndexoldIndex

transforms.conf looks like this:

[overrideIndexoldIndex]
DEST_KEY =_MetaData:Index
REGEX = oldIndex
SOURCE_KEY=_MetaData:Index
FORMAT = newIndex

My understanding is that it is applying this transform for all data from host:MM[0-9]{6}-PC. The transform is just redirecting index:oldIndex to newIndex. There is a lot of data from hosts that matches this criteria. Is there a way to first check that the index is oldIndex and than look for those hosts and apply the transform then. Logically this would increase performance as there is far less data being sent to the index oldIndex than, the data being sent from those hosts that match our criteria.

Essentially I would like to understand the parsing of data better surrounding transforms and if this is a valid optimization how to go about implementing it.

Tags (1)
0 Karma

Anam
Community Manager
Community Manager

Hi @m91886

Looks like you have a few possible solutions to your question. If one of them provided a working solution, please don't forget to click "Accept" below the best answer to resolve this post. If you still need help, please leave a comment. Don’t forget to upvote anything that was helpful too. Thanks!

0 Karma

somesoni2
Revered Legend

For sourcetypes, you don't need the prefix. The stanza name should be like this

[source::<source>] OR [host::<host>] OR  [<sourcetype>] 
0 Karma

woodcock
Esteemed Legend

Your logic and configurations are correct and it cannot be done any other way, other than by source or by sourcetype instead of by host in props.conf.

0 Karma

somesoni2
Revered Legend

You current configuration (assuming it's been placed on the instance that does the parsing i.e. heavy forwarder or indexer whichever comes first) override index name to newIndex for each event tagged with index=oldIndex and coming from hosts matching pattern MM[0-9]{6}-PC. Unfortunately, this override can only be setup at sourcetype, source OR host level, and not at index level.

Any specific reasons for overriding index for those host/index combination? Could you explain your requirement little more in detail?

0 Karma

m91886
New Member

Actually, the sourcetype would work. The reason is that a group is sending splunk logs to two environments. On one those environment's logs are set for oldIndex and on the other environment we want them in newIndex. Since the universal forwarder only sends those logs using one index to both environments we are using the transforms to change the index in our environment.

Would this be valid?

[sourcetype:OriginalSourceType]
[host::MM[0-9]{6}-PC]
TRANSFORMS-index = overrideIndexoldIndex

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...