topic Re: How to reroute index for props/transform in Getting Data In

How to reroute index for props/transform

dglass0215 — Mon, 26 Aug 2019 15:00:22 GMT

Hello,

I have my props/transforms setup so that it routes data to specific indexes (For the most part) based on hostname. This was working great until I realized that it was also including data that was meant for internal indexes like _internal. Specifically for splunkd sourcetype. This was making my license utilize way more data than it should have been. So I added a prop for [splunkd] sourcetype to route data to _internal. This is now working for alot of my hosts but for some reason there are still 4 indexes receiving data that should be routed to _internal, they are indexes named mhh, irw, lvn, lee. Any help is much appreciated. Props and transforms below:

PROPS:

[HawkeyeIOLS]
TRANSFORMS-set= setnull,setparsing
SEDCMD-DOB = s#DOB/\d{8}#DOB/[REDACTED]#g
SEDCMD-SEX = s#SEX/.#SEX/[REDACTED]#g
SEDCMD-RAC = s#RAC/.#RAC/[REDACTED]#g

[splunkd]
TRANSFORMS-set_index_internal = set_index_internal
priority=10

[host::(5050-LANE|5050-RGWS)]
TRANSFORMS-set_index_5050 = set_index_bel

[host::irw*]
TRANSFORMS-set_index_irw = set_index_irw

[host::mhh*]
TRANSFORMS-set_index_mhh = set_index_mhh

[host::mcn*]
TRANSFORMS-set_index_mcn = set_index_mcn

[host::lee*]
TRANSFORMS-set_index_lee = set_index_lee

[host::lvn*]
TRANSFORMS-set_index_lvn = set_index_lvn

[host::riv*]
TRANSFORMS-set_index_riv = set_index_riv

[host::con*]
TRANSFORMS-set_index_con = set_index_con

[host::ann*]
TRANSFORMS-set_index_ann = set_index_ann

[host::bel*]
TRANSFORMS-set_index_bel = set_index_bel

[host::apg*]
TRANSFORMS-set_index_apg = set_index_apg

[host::hol*]
TRANSFORMS-set_index_hol = set_index_hol

TRANSFORMS:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \b(?:offline|online|HTTP 502 Bad Gateway|CheckUpdate: Error)\b
DEST_KEY = queue
FORMAT = indexQueue

[set_index_internal]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = _internal

[set_index_con]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = motco

[set_index_ann]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = ANN

[set_index_bel]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = BEL

[set_index_apg]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = APG

[set_index_hol]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = HOL

[set_index_irw]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = IRW

[set_index_mhh]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = mhh

[set_index_mcn]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = mcn

[set_index_lee]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = lee

[set_index_lvn]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = lvn

[set_index_riv]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = riv

Re: How to reroute index for props/transform

DavidHourani — Mon, 26 Aug 2019 16:29:06 GMT

Hi @dglass0215,

You can improve your config as follows to avoid any "conditions" causing data to escape and go in the wrong index.

To do so, instead of capturing the sourcetype splunkd as you did from one side and the hosts from the other, you can do both in the same stanza. Below an example for a single host :

[host::lee*]
TRANSFORMS-set_index_lee = set_index_lee, set_index_internal

[set_index_lee]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = lee

[set_index_internal]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = _MetaData:Index
REGEX =  splunkd
FORMAT = _internal

The transforms need to go in the order shown above (set_index_lee, set_index_internal) to be sure the indexes get changed as needed.

Once the above is done for all your hosts you can get rid of the config below from props.conf :

[splunkd]
TRANSFORMS-set_index_internal = set_index_internal
priority=10

Let me know how that works for you.

Cheers,
David

Re: How to reroute index for props/transform

dglass0215 — Mon, 26 Aug 2019 18:07:09 GMT

Thanks! This definitely works. Still would like to know what was wrong with what I originally had but at least this works!

Re: How to reroute index for props/transform

DavidHourani — Mon, 26 Aug 2019 19:08:22 GMT

You're welcome !

And yeah I would like to know the problem too... you could test it out by simplifying the set up as much as possible. Maybe remove all the stanzas keeping only one of the ones causing the issue and the splunkd one, then see if thats still not working..