Getting Data In

How to reroute index for props/transform

dglass0215
Path Finder

Hello,

I have my props/transforms setup so that it routes data to specific indexes (For the most part) based on hostname. This was working great until I realized that it was also including data that was meant for internal indexes like _internal. Specifically for splunkd sourcetype. This was making my license utilize way more data than it should have been. So I added a prop for [splunkd] sourcetype to route data to _internal. This is now working for alot of my hosts but for some reason there are still 4 indexes receiving data that should be routed to _internal, they are indexes named mhh, irw, lvn, lee. Any help is much appreciated. Props and transforms below:

PROPS:

[HawkeyeIOLS]
TRANSFORMS-set= setnull,setparsing
SEDCMD-DOB = s#DOB/\d{8}#DOB/[REDACTED]#g
SEDCMD-SEX = s#SEX/.#SEX/[REDACTED]#g
SEDCMD-RAC = s#RAC/.#RAC/[REDACTED]#g

[splunkd]
TRANSFORMS-set_index_internal = set_index_internal
priority=10

[host::(5050-LANE|5050-RGWS)]
TRANSFORMS-set_index_5050 = set_index_bel

[host::irw*]
TRANSFORMS-set_index_irw = set_index_irw

[host::mhh*]
TRANSFORMS-set_index_mhh = set_index_mhh

[host::mcn*]
TRANSFORMS-set_index_mcn = set_index_mcn

[host::lee*]
TRANSFORMS-set_index_lee = set_index_lee

[host::lvn*]
TRANSFORMS-set_index_lvn = set_index_lvn

[host::riv*]
TRANSFORMS-set_index_riv = set_index_riv

[host::con*]
TRANSFORMS-set_index_con = set_index_con

[host::ann*]
TRANSFORMS-set_index_ann = set_index_ann

[host::bel*]
TRANSFORMS-set_index_bel = set_index_bel

[host::apg*]
TRANSFORMS-set_index_apg = set_index_apg

[host::hol*]
TRANSFORMS-set_index_hol = set_index_hol

TRANSFORMS:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \b(?:offline|online|HTTP 502 Bad Gateway|CheckUpdate: Error)\b
DEST_KEY = queue
FORMAT = indexQueue

[set_index_internal]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = _internal

[set_index_con]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = motco

[set_index_ann]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = ANN

[set_index_bel]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = BEL

[set_index_apg]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = APG

[set_index_hol]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = HOL

[set_index_irw]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = IRW

[set_index_mhh]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = mhh

[set_index_mcn]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = mcn

[set_index_lee]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = lee

[set_index_lvn]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = lvn

[set_index_riv]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = riv
0 Karma
1 Solution

DavidHourani
Super Champion

Hi @dglass0215,

You can improve your config as follows to avoid any "conditions" causing data to escape and go in the wrong index.

To do so, instead of capturing the sourcetype splunkd as you did from one side and the hosts from the other, you can do both in the same stanza. Below an example for a single host :

[host::lee*]
TRANSFORMS-set_index_lee = set_index_lee, set_index_internal

[set_index_lee]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = lee

[set_index_internal]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = _MetaData:Index
REGEX =  splunkd
FORMAT = _internal

The transforms need to go in the order shown above (set_index_lee, set_index_internal) to be sure the indexes get changed as needed.

Once the above is done for all your hosts you can get rid of the config below from props.conf :

[splunkd]
TRANSFORMS-set_index_internal = set_index_internal
priority=10

Let me know how that works for you.

Cheers,
David

View solution in original post

DavidHourani
Super Champion

Hi @dglass0215,

You can improve your config as follows to avoid any "conditions" causing data to escape and go in the wrong index.

To do so, instead of capturing the sourcetype splunkd as you did from one side and the hosts from the other, you can do both in the same stanza. Below an example for a single host :

[host::lee*]
TRANSFORMS-set_index_lee = set_index_lee, set_index_internal

[set_index_lee]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = lee

[set_index_internal]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = _MetaData:Index
REGEX =  splunkd
FORMAT = _internal

The transforms need to go in the order shown above (set_index_lee, set_index_internal) to be sure the indexes get changed as needed.

Once the above is done for all your hosts you can get rid of the config below from props.conf :

[splunkd]
TRANSFORMS-set_index_internal = set_index_internal
priority=10

Let me know how that works for you.

Cheers,
David

dglass0215
Path Finder

Thanks! This definitely works. Still would like to know what was wrong with what I originally had but at least this works!

0 Karma

DavidHourani
Super Champion

You're welcome !

And yeah I would like to know the problem too... you could test it out by simplifying the set up as much as possible. Maybe remove all the stanzas keeping only one of the ones causing the issue and the splunkd one, then see if thats still not working..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...