Filtering WMI events on heavy forwarder

[PickleRick]

Yes, I know that filtering was discussed many times here but my case is slightly different.

I have a UF pulling events using WMI. It then pushes the events to upstream HFs.

On the HFs I tried to do filtering similarily to https://docs.splunk.com/Documentation/Splunk/8.2.2/Forwarding/Routeandfilterdatad#Filter_WMI_and_Eve...

But in my case there are two differences:

1) I didn't want to filter out particular events. I wanted to filter out everything, just keep two kinds of events. Seems easy - just do a default transform with REGEX=. to set queue to nullQueue and then just match the ones you want to index and set queue to indexQueue. Well, it doesn't work. Maybe because:

2) I didn't want to apply this to the whole sourcetype. And here's where I suspect something might have gone wrong because if it was just that my transforms are bad, the default one sending to the nullQueue should work. But it seems that they don't work at all.

My definitions:

props.conf:

[host::TEST...]
TRANSFORMS-routing = TEST_default_drop,TEST_index

(the hosts I'm getting the data from are called TEST01.domain.com, TEST02.domain.com and so on; I already tried host::TEST*.domain.com)

transforms.conf

[TEST_default_drop]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

[TEST_index]
REGEX=(?m)^EventCode=(103|104)
DEST_KEY=queue
FORMAT=indexQueue

 Everything seems to be reasonably well, but it doesn't work - I'm getting all data in my index - no filtering at all.

I wouldn't want to configure filtering for the whole sourcetype because I might use WMI in the future for other things and this particular filtering is only for this one kind of sources.

Does UF set host field to something other than I'm expecting? Can I debug it somehow reasonably? (the UF is kinda a production one in general so it'd not be wise to turn fully blown debug of everything on it on)

[PickleRick · Accepted solution]

OK. It seems that - as I somehow suspected - the issue was with the host field.

Apparently the host field is getting ingested as the hostname of the UF host (since I'm using the same host for other sources I couldn't just filter all events by this hostname), not as the source systems hostnames. Apparently the TA for Windows app rewrites the host field in search time.

So I had to define props for whole WMI:WinEventLog:Application sourcetype and do some ridiculous regex matching like:

[SPE_default_drop]
REGEX=(?m)^ComputerName=TEST\d{2}.domain.com
DEST_KEY=queue
FORMAT=nullQueue

[SPE_index]
REGEX=(?s)^EventCode=(103|104).*ComputerName=TEST\d{2}.domain.com
DEST_KEY=queue
FORMAT=indexQueue

Luckily, the WMI-pulled events seem to have the fields always in the same order so I can match EventCode.*ComputerName

[PickleRick]

Hmm... interesting. Because the regex seems to be matching properly i regex101 but doesn't seem to match on the HF. Needs investigating.