Getting Data In

Rename Index based on Host AND Index name

DavidHourani
Super Champion

Hello Splunkers,

I have multiple sources sending each data for multiple indexes towards on central Universal Forwarder. I need to rename some indexes based on the host and on the index name.

For example :
If host is A and index name is X then rename index to Y
If host is A and index name is W then keep it W
If host is B and index name is X keep it X
If host is B and index name is W rename to Y

Can you guys help out please ?

Regards,
David

0 Karma
1 Solution

somesoni2
Revered Legend

On your indexer or heavy forwarder:

 transforms.conf 
 [overrideindexhostA]
 DEST_KEY =_MetaData:Index
 REGEX = X
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y

 [overrideindexhostB]
 DEST_KEY =_MetaData:Index
 REGEX = W
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y



 #props.conf 
 [host::A]
 TRANSFORMS-index = overrideindexhostA

 [host::B]
 TRANSFORMS-index = overrideindexhostB

View solution in original post

jkat54
SplunkTrust
SplunkTrust
Props.conf
[sourcetypeName]
TRANSFORM-indexA=indexA
TRANSFORM-indexB=indexB

Transforms.conf
[indexA]
SOURCE_KEY=MetaData:Host
REGEX=(Regular expression that matches hosts)
FORMAT=indexNameA
DEST_KEY=_MetaData:Index

[indexB]
SOURCE_KEY=MetaData:Host
REGEX=(Regular expression that matches hosts)
FORMAT=indexNameB
DEST_KEY=_MetaData:Index

jkat54
SplunkTrust
SplunkTrust
 It uses match expressions which is virtually same as PCRE except *, ..., and .  

 . matches a .

DavidHourani
Super Champion

also... Your configuration worked on a standalone lab but in a clustered environment it's not working... Any idea why ? I can't understand what's going wrong...

0 Karma

jkat54
SplunkTrust
SplunkTrust

It has to go on the forwarders.

DavidHourani
Super Champion

I can't do anything on the forwarders... they are kind of blocked appliances..

0 Karma

DavidHourani
Super Champion

what about the handling dots in the host stanza ? for example --> [host::bla.bla.bla] does that work or should I write [host::bla.bla.bla] ?

0 Karma

somesoni2
Revered Legend

On your indexer or heavy forwarder:

 transforms.conf 
 [overrideindexhostA]
 DEST_KEY =_MetaData:Index
 REGEX = X
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y

 [overrideindexhostB]
 DEST_KEY =_MetaData:Index
 REGEX = W
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y



 #props.conf 
 [host::A]
 TRANSFORMS-index = overrideindexhostA

 [host::B]
 TRANSFORMS-index = overrideindexhostB

DavidHourani
Super Champion

one more thing, if my host contains . for example "blabla.com" should i write : [host::blabla.com] or [host::blabla.com] ?

0 Karma

somesoni2
Revered Legend

If your host name ends with blahblah.com, then use [host::*blahblah.com]

0 Karma

DavidHourani
Super Champion

yeah I mean how do you handle the dot ? should you escape it . or just write it as is ? Anddddd btw you should swap the src and dst keys, src comes first ^^

0 Karma

somesoni2
Revered Legend

No not required. Stanza names are not pure regex format, they accept wildcard and dot is treated as regular character.

0 Karma

DavidHourani
Super Champion

why does this work on a standalone indexer but not on indexers in a clustered environment ?

0 Karma

DavidHourani
Super Champion

you rock man 🙂

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Assuming you mean that, at index time, you want to override the index for the data being indexed.

See YannK's answer here for the general method -
https://answers.splunk.com/answers/52198/change-sourcetype-index-after-data-is-indexed-from-forwarde...

It's a bit annoying, but complex calculation is not really available at index time, other than regexes.

For this, you'd want to set up a rule for Host A and a second rule for host B. Establish one regex for each host that seds the entire old index name to the entire new index name. If it does not fully match, then no change should occur.

somesoni2's answer gives the exact code. Note that in these stanzas, the sed is broken up into two pieces - the match pattern, with goes under REGEX=, and the replacement pattern, which goes under FORMAT=.

DalJeanis
SplunkTrust
SplunkTrust

Dang, I need to move this to a comment on somesoni2's post, which provides the exact code.

In these stanzas, the regex is broken up into old (which goes into REGEX 😃 and new (which goes into FORMAT = )

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

By "rename index", I assume you mean "assign to different index at index time"?

0 Karma

DavidHourani
Super Champion

yeap exactly

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...