topic Re: Rename Index based on Host AND Index name in Getting Data In

Rename Index based on Host AND Index name

DavidHourani — Wed, 01 Mar 2017 14:50:15 GMT

Hello Splunkers,

I have multiple sources sending each data for multiple indexes towards on central Universal Forwarder. I need to rename some indexes based on the host and on the index name.

For example :
If host is A and index name is X then rename index to Y
If host is A and index name is W then keep it W
If host is B and index name is X keep it X
If host is B and index name is W rename to Y

Can you guys help out please ?

Regards,
David

Re: Rename Index based on Host AND Index name

DalJeanis — Wed, 01 Mar 2017 15:28:17 GMT

By "rename index", I assume you mean "assign to different index at index time"?

Re: Rename Index based on Host AND Index name

somesoni2 — Wed, 01 Mar 2017 15:34:13 GMT

On your indexer or heavy forwarder:

 transforms.conf 
 [overrideindexhostA]
 DEST_KEY =_MetaData:Index
 REGEX = X
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y

 [overrideindexhostB]
 DEST_KEY =_MetaData:Index
 REGEX = W
 SOURCE_KEY=_MetaData:Index
 FORMAT = Y



 #props.conf 
 [host::A]
 TRANSFORMS-index = overrideindexhostA

 [host::B]
 TRANSFORMS-index = overrideindexhostB

Re: Rename Index based on Host AND Index name

DalJeanis — Wed, 01 Mar 2017 15:34:14 GMT

Assuming you mean that, at index time, you want to override the index for the data being indexed.

See YannK's answer here for the general method -
https://answers.splunk.com/answers/52198/change-sourcetype-index-after-data-is-indexed-from-forwarder.html

It's a bit annoying, but complex calculation is not really available at index time, other than regexes.

For this, you'd want to set up a rule for Host A and a second rule for host B. Establish one regex for each host that seds the entire old index name to the entire new index name. If it does not fully match, then no change should occur.

somesoni2's answer gives the exact code. Note that in these stanzas, the sed is broken up into two pieces - the match pattern, with goes under REGEX=, and the replacement pattern, which goes under FORMAT=.

Re: Rename Index based on Host AND Index name

DavidHourani — Wed, 01 Mar 2017 15:36:28 GMT

you rock man 🙂

Re: Rename Index based on Host AND Index name

DavidHourani — Wed, 01 Mar 2017 15:37:07 GMT

yeap exactly

Re: Rename Index based on Host AND Index name

jkat54 — Wed, 01 Mar 2017 15:39:24 GMT

Props.conf
[sourcetypeName]
TRANSFORM-indexA=indexA
TRANSFORM-indexB=indexB

Transforms.conf
[indexA]
SOURCE_KEY=MetaData:Host
REGEX=(Regular expression that matches hosts)
FORMAT=indexNameA
DEST_KEY=_MetaData:Index

[indexB]
SOURCE_KEY=MetaData:Host
REGEX=(Regular expression that matches hosts)
FORMAT=indexNameB
DEST_KEY=_MetaData:Index

Re: Rename Index based on Host AND Index name

DalJeanis — Wed, 01 Mar 2017 15:52:01 GMT

Dang, I need to move this to a comment on somesoni2's post, which provides the exact code.

In these stanzas, the regex is broken up into old (which goes into REGEX 😃 and new (which goes into FORMAT = )

Re: Rename Index based on Host AND Index name

DavidHourani — Mon, 06 Mar 2017 12:23:06 GMT

one more thing, if my host contains . for example "blabla.com" should i write : [host::blabla.com] or [host::blabla.com] ?

Re: Rename Index based on Host AND Index name

somesoni2 — Mon, 06 Mar 2017 15:09:16 GMT

If your host name ends with blahblah.com, then use [host::*blahblah.com]

Re: Rename Index based on Host AND Index name

DavidHourani — Tue, 07 Mar 2017 10:26:44 GMT

yeah I mean how do you handle the dot ? should you escape it . or just write it as is ? Anddddd btw you should swap the src and dst keys, src comes first ^^

Re: Rename Index based on Host AND Index name

DavidHourani — Tue, 07 Mar 2017 10:27:51 GMT

what about the handling dots in the host stanza ? for example --> [host::bla.bla.bla] does that work or should I write [host::bla.bla.bla] ?

Re: Rename Index based on Host AND Index name

jkat54 — Tue, 07 Mar 2017 11:35:58 GMT

 It uses match expressions which is virtually same as PCRE except *, ..., and .  

 . matches a .

Re: Rename Index based on Host AND Index name

DavidHourani — Tue, 07 Mar 2017 14:41:11 GMT

also... Your configuration worked on a standalone lab but in a clustered environment it's not working... Any idea why ? I can't understand what's going wrong...

Re: Rename Index based on Host AND Index name

somesoni2 — Tue, 07 Mar 2017 15:04:39 GMT

No not required. Stanza names are not pure regex format, they accept wildcard and dot is treated as regular character.

Re: Rename Index based on Host AND Index name

jkat54 — Tue, 07 Mar 2017 21:44:57 GMT

It has to go on the forwarders.

Re: Rename Index based on Host AND Index name

DavidHourani — Mon, 10 Apr 2017 11:03:24 GMT

why does this work on a standalone indexer but not on indexers in a clustered environment ?

Re: Rename Index based on Host AND Index name

DavidHourani — Mon, 10 Apr 2017 11:03:48 GMT

I can't do anything on the forwarders... they are kind of blocked appliances..