Rename Index based on Host AND Index name

[DavidHourani]

Hello Splunkers,

I have multiple sources sending each data for multiple indexes towards on central Universal Forwarder. I need to rename some indexes based on the host and on the index name.

For example :
If host is A and index name is X then rename index to Y
If host is A and index name is W then keep it W
If host is B and index name is X keep it X
If host is B and index name is W rename to Y

Can you guys help out please ?

Regards,
David

[jkat54]

Props.conf
[sourcetypeName]
TRANSFORM-indexA=indexA
TRANSFORM-indexB=indexB

Transforms.conf
[indexA]
SOURCE_KEY=MetaData:Host
REGEX=(Regular expression that matches hosts)
FORMAT=indexNameA
DEST_KEY=_MetaData:Index

[indexB]
SOURCE_KEY=MetaData:Host
REGEX=(Regular expression that matches hosts)
FORMAT=indexNameB
DEST_KEY=_MetaData:Index

[jkat54]

 It uses match expressions which is virtually same as PCRE except *, ..., and .  

 . matches a .

[DavidHourani]

also... Your configuration worked on a standalone lab but in a clustered environment it's not working... Any idea why ? I can't understand what's going wrong...

[jkat54]

It has to go on the forwarders.

[DavidHourani]

I can't do anything on the forwarders... they are kind of blocked appliances..

[DavidHourani]

what about the handling dots in the host stanza ? for example --> [host::bla.bla.bla] does that work or should I write [host::bla.bla.bla] ?

[somesoni2 · Accepted solution]

On your indexer or heavy forwarder:

 transforms.conf 
 [overrideindexhostA]
 DEST_KEY =_MetaData:Index
 REGEX = X
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y

 [overrideindexhostB]
 DEST_KEY =_MetaData:Index
 REGEX = W
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y



 #props.conf 
 [host::A]
 TRANSFORMS-index = overrideindexhostA

 [host::B]
 TRANSFORMS-index = overrideindexhostB

[DavidHourani]

one more thing, if my host contains . for example "blabla.com" should i write : [host::blabla.com] or [host::blabla.com] ?

[somesoni2]

If your host name ends with blahblah.com, then use [host::*blahblah.com]

[DavidHourani]

yeah I mean how do you handle the dot ? should you escape it . or just write it as is ? Anddddd btw you should swap the src and dst keys, src comes first ^^

[somesoni2]

No not required. Stanza names are not pure regex format, they accept wildcard and dot is treated as regular character.

[DavidHourani]

why does this work on a standalone indexer but not on indexers in a clustered environment ?

[DavidHourani]

you rock man 🙂

[DalJeanis]

Assuming you mean that, at index time, you want to override the index for the data being indexed.

See YannK's answer here for the general method -
https://answers.splunk.com/answers/52198/change-sourcetype-index-after-data-is-indexed-from-forwarde...

It's a bit annoying, but complex calculation is not really available at index time, other than regexes.

For this, you'd want to set up a rule for Host A and a second rule for host B. Establish one regex for each host that seds the entire old index name to the entire new index name. If it does not fully match, then no change should occur.

somesoni2's answer gives the exact code. Note that in these stanzas, the sed is broken up into two pieces - the match pattern, with goes under REGEX=, and the replacement pattern, which goes under FORMAT=.

[DalJeanis]

Dang, I need to move this to a comment on somesoni2's post, which provides the exact code.

In these stanzas, the regex is broken up into old (which goes into REGEX 😃 and new (which goes into FORMAT = )

[DalJeanis]

By "rename index", I assume you mean "assign to different index at index time"?

[DavidHourani]

yeap exactly