Getting Data In

Rename Index based on Host AND Index name

SplunkTrust
SplunkTrust

Hello Splunkers,

I have multiple sources sending each data for multiple indexes towards on central Universal Forwarder. I need to rename some indexes based on the host and on the index name.

For example :
If host is A and index name is X then rename index to Y
If host is A and index name is W then keep it W
If host is B and index name is X keep it X
If host is B and index name is W rename to Y

Can you guys help out please ?

Regards,
David

0 Karma
1 Solution

SplunkTrust
SplunkTrust

On your indexer or heavy forwarder:

 transforms.conf 
 [overrideindexhostA]
 DEST_KEY =_MetaData:Index
 REGEX = X
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y

 [overrideindexhostB]
 DEST_KEY =_MetaData:Index
 REGEX = W
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y



 #props.conf 
 [host::A]
 TRANSFORMS-index = overrideindexhostA

 [host::B]
 TRANSFORMS-index = overrideindexhostB

View solution in original post

SplunkTrust
SplunkTrust
Props.conf
[sourcetypeName]
TRANSFORM-indexA=indexA
TRANSFORM-indexB=indexB

Transforms.conf
[indexA]
SOURCE_KEY=MetaData:Host
REGEX=(Regular expression that matches hosts)
FORMAT=indexNameA
DEST_KEY=_MetaData:Index

[indexB]
SOURCE_KEY=MetaData:Host
REGEX=(Regular expression that matches hosts)
FORMAT=indexNameB
DEST_KEY=_MetaData:Index

SplunkTrust
SplunkTrust
 It uses match expressions which is virtually same as PCRE except *, ..., and .  

 . matches a .

SplunkTrust
SplunkTrust

also... Your configuration worked on a standalone lab but in a clustered environment it's not working... Any idea why ? I can't understand what's going wrong...

0 Karma

SplunkTrust
SplunkTrust

It has to go on the forwarders.

SplunkTrust
SplunkTrust

I can't do anything on the forwarders... they are kind of blocked appliances..

0 Karma

SplunkTrust
SplunkTrust

what about the handling dots in the host stanza ? for example --> [host::bla.bla.bla] does that work or should I write [host::bla.bla.bla] ?

0 Karma

SplunkTrust
SplunkTrust

On your indexer or heavy forwarder:

 transforms.conf 
 [overrideindexhostA]
 DEST_KEY =_MetaData:Index
 REGEX = X
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y

 [overrideindexhostB]
 DEST_KEY =_MetaData:Index
 REGEX = W
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y



 #props.conf 
 [host::A]
 TRANSFORMS-index = overrideindexhostA

 [host::B]
 TRANSFORMS-index = overrideindexhostB

View solution in original post

SplunkTrust
SplunkTrust

one more thing, if my host contains . for example "blabla.com" should i write : [host::blabla.com] or [host::blabla.com] ?

0 Karma

SplunkTrust
SplunkTrust

If your host name ends with blahblah.com, then use [host::*blahblah.com]

0 Karma

SplunkTrust
SplunkTrust

yeah I mean how do you handle the dot ? should you escape it . or just write it as is ? Anddddd btw you should swap the src and dst keys, src comes first ^^

0 Karma

SplunkTrust
SplunkTrust

No not required. Stanza names are not pure regex format, they accept wildcard and dot is treated as regular character.

0 Karma

SplunkTrust
SplunkTrust

why does this work on a standalone indexer but not on indexers in a clustered environment ?

0 Karma

SplunkTrust
SplunkTrust

you rock man 🙂

0 Karma

SplunkTrust
SplunkTrust

Assuming you mean that, at index time, you want to override the index for the data being indexed.

See YannK's answer here for the general method -
https://answers.splunk.com/answers/52198/change-sourcetype-index-after-data-is-indexed-from-forwarde...

It's a bit annoying, but complex calculation is not really available at index time, other than regexes.

For this, you'd want to set up a rule for Host A and a second rule for host B. Establish one regex for each host that seds the entire old index name to the entire new index name. If it does not fully match, then no change should occur.

somesoni2's answer gives the exact code. Note that in these stanzas, the sed is broken up into two pieces - the match pattern, with goes under REGEX=, and the replacement pattern, which goes under FORMAT=.

SplunkTrust
SplunkTrust

Dang, I need to move this to a comment on somesoni2's post, which provides the exact code.

In these stanzas, the regex is broken up into old (which goes into REGEX 😃 and new (which goes into FORMAT = )

0 Karma

SplunkTrust
SplunkTrust

By "rename index", I assume you mean "assign to different index at index time"?

0 Karma

SplunkTrust
SplunkTrust

yeap exactly

0 Karma