topic Re: Can I still send data to nullQueue while using _MetaData:Index to send data to an Alternate Index? in Getting Data In

Can I still send data to nullQueue while using _MetaData:Index to send data to an Alternate Index?

mmcgrew — Wed, 23 Aug 2017 17:25:53 GMT

I have one source directory in the inputs.conf file that I need to parse out and send different events to different Indexes.
I attempt to do this by using the _Metadata:Index Key within the transforms.conf file. While this works, ( I can direct events to different indexes using _Metadata:Index), I cannot send my unwanted items to nullQueue. The problem this creates is all the leftover events (that didn't match a regex statement I had in transforms.conf) gets sent to the default Index I defined in inputs.conf. I should be able to send all items that don't match my regex statement to nullQueue but I cannot do this while utilizing the _Metadata:Index Key in transforms.conf....looking below at the examples I've provided, once I use the "TRANSFORMS-null=setnull" in the props.conf file - nothing works (none of my data ends up in any index). I have tried moving the "TRANSFORMS-null=setnull" around to the top and bottom of the props file but still get nothing...once I remove the "TRANSFORMS-null=setnull" statement from the props.conf file, I get the data I want in index2 and index3 but then get all the leftover data I don't want in index1. It seems as though whatever index is defined in the inputs.conf file will get all the leftover items as it is interpreted as the default index. If I could use nullQueue, then everything would be good. Any suggestions or help is appreciated. Thanks.

inputs.conf
[monitor:///opt/sdata/events/server-A/splunk*]
host = server-A
source = A
sourcetype = data1
index = index1
disabled = 0

props.conf
[host::server-A]
TRANSFORMS-set1 = setparsingdata1
[host::server-A]
TRANSFORMS-set2 = setparsingdata2
[host::server-A]
TRANSFORMS-null = setnull

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsingdata1]
REGEX = ((^(?=.Summary="BGP Peer Connection Established.").+))
DEST_KEY = _MetaData:Index
FORMAT = index2

[setparsingdata2]
REGEX = ((^(?=.Summary="BGP Peer Connection Idle.").+))
DEST_KEY = _MetaData:Index
FORMAT = index3

Re: Can I still send data to nullQueue while using _MetaData:Index to send data to an Alternate Index?

somesoni2 — Wed, 23 Aug 2017 17:48:08 GMT

Give this a try [untested, to try on some sandbox first) (no changes to inputs.conf)

props.conf (make sure to keep the setnull as leftmost entry)

[host::server-A]
TRANSFORMS-set1 = setnull,setparsingdata1,setparsingdata2

Updated
transforms.conf

[setnull]
REGEX = ^(?!.Summary=\"BGP Peer Connection (Established|Idle).)
DEST_KEY = queue
FORMAT = nullQueue

[setparsingdata1]
REGEX = ((^(?=.Summary="BGP Peer Connection Established.").+))
DEST_KEY = _MetaData:Index
FORMAT = index2

[setparsingdata2]
REGEX = ((^(?=.Summary="BGP Peer Connection Idle.").+))
DEST_KEY = _MetaData:Index
FORMAT = index3

Re: Can I still send data to nullQueue while using _MetaData:Index to send data to an Alternate Index?

mmcgrew — Wed, 23 Aug 2017 18:01:16 GMT

Thank you for the quick response. I just tried this but no luck...no data getting to any of the indexes. Once I remove the setnull...it works - but then I end up indexing a bunch of unwanted events.

Re: Can I still send data to nullQueue while using _MetaData:Index to send data to an Alternate Index?

somesoni2 — Wed, 23 Aug 2017 19:30:21 GMT

Try the updated answer.

Re: Can I still send data to nullQueue while using _MetaData:Index to send data to an Alternate Index?

mmcgrew — Wed, 23 Aug 2017 20:05:59 GMT

I have tried this method that you suggested but it did not work either. It does not seem that the REGEX statement within the setnull stanza is the problem....regardless of what I change the REGEX to I cannot seem to get the data to parse to any index while using the setnull/nullQueue option...which is why it seems like there is a conflict with using nullQueue and _Metadata:Index because either of those work independently of each other but when used together it doesn't work.