topic Re: Route and filter data in Getting Data In

Is it possible to filter specific field values in indexers without HeavyForwarder in indexer cluster ?

khanlarloo — Mon, 04 Apr 2022 14:42:53 GMT

Is it possible to filter specific field values in indexers without HeavyForwarder in indexer cluster ?

Re: Route and filter data

gcusello — Sun, 03 Apr 2022 10:02:51 GMT

filtering is possible only on Indexers (also clustered) or (when present) on Heavy Forwarders, it isn't possible (except Wineventlogs) on Universal Forwarders. you have to filter data on the first full Splunk instance the data encounter

In other words: if you have HFs in your architecture you have to filter data on HFs, otherwise on Indexers, it isn't possible to filer data on Indexers when you also have HFs.

But you want to filter the full events or a part of them?

If full events, you can see how to do it at https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad

if a part of them, you can see how to do it at https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Anonymizedata

Ciao.

Giuseppe

Re: Route and filter data

khanlarloo — Sun, 03 Apr 2022 10:32:17 GMT

Thank You for answering my question, Then How should I configure the output.conf?

we have 60 indexer, and 5 Sh, and 500 UF with CM

Re: Route and filter data

gcusello — Sun, 03 Apr 2022 10:47:07 GMT

Hi @khanlarloo,

outputs.conf isn't useful in events filtering, but it's useful only to configure your UFs to send logs to Indexers.

To filter events, you have to configure props.conf and transforms.conf on you Indexers, as described in the above links.

Ciao.

Giuseppe

Re: Route and filter data

PickleRick — Sun, 03 Apr 2022 11:46:30 GMT

Ok, let me rephrase the @gcusello 's answer a little.

Routing and filtering is something that is a part of index-time event processing and it happens on the first "heavy" (non-UF) splunk component in event's path. So if you send straight from UF's to indexers, it's gonna happen on indexers. If you have HF's in front of indexers you'd do it on HF's but if you don't, do it on indexers because they are the first "full" or "heavy" component.

What you should do is create a props and transforms settings according to https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad pack it into an app and push to your indexers as you'd do with any other app.

Re: Route and filter data

khanlarloo — Mon, 11 Apr 2022 06:43:45 GMT

It doesn't work.

my props.conf

[XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
TRANSFORMS-null= ConfigSync-win

transforms.conf

[ConfigSync-win]
REGEX = Path=\"C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\"
DEST_KEY = queue
FORMAT = nullQueue

Re: Route and filter data

gcusello — Mon, 11 Apr 2022 06:56:29 GMT

Hi @khanlarloo,

are you sure that "XmlWinEventLog:Microsoft-Windows-PowerShell/Operational" is the sourcetype of your flow?

did you checked the regex in transforms.conf?

Then, where do you located props.conf and transforms.conf?

they must be located on Indexers or (when present) on Heavy Forwarders.

Ciao.

Giuseppe

Re: Route and filter data

khanlarloo — Mon, 11 Apr 2022 07:37:15 GMT

"XmlWinEventLog:Microsoft-Windows-PowerShell/Operational" is source, ilocated transforms.conf and props.con in indexeres under /$SPLUNK_HOME$/etc/slave-apps/_cluster/local

Re: Route and filter data

gcusello — Mon, 11 Apr 2022 07:41:21 GMT

Hi @khanlarloo,

in this case, the room name must be:

[source::XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]

if you don't use source or host, the name is always the sourcetype.

Ciao.

Giuseppe

Re: Route and filter data

khanlarloo — Mon, 11 Apr 2022 10:29:56 GMT

my raw log is :

</Data><Data Name='ScriptBlockId'>dd493fb0-9281-4197-a347-a7afaeea3632</Data><Data Name='Path'>C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1</Data></EventData></Event>

and i want to filter this part "C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1"

can you tell me what is the regex for it?

Re: Route and filter data

gcusello — Mon, 11 Apr 2022 11:39:14 GMT

Hi @khanlarloo,

this is your regex

C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\<

that you can test at https://regex101.com/r/loRC2Y/1

Ciao.

Giuseppe

Re: Route and filter data

PickleRick — Mon, 11 Apr 2022 11:47:39 GMT

Wait, wait, wait...

Your raw data is like this:

<Data Name='Path'>C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1</Data>

and your regex is

Path=\"C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\"

You don't have string fitting that regex in your event. You're trying to do some form of field value matching which will not work instead of regex match.

Try something like

'Path'>C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync.ps1<

As @gcusello said - use https://regex101.com to verify your regex against your data

Re: Route and filter data

khanlarloo — Wed, 18 May 2022 05:26:50 GMT

i do it, but i still recieve this log.

transforms.conf

[setnull]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = (?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1

props.conf

[source::XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
TRANSFORMS-null= setnull

Re: Route and filter data

gcusello — Wed, 18 May 2022 06:37:15 GMT

Hi @khanlarloo,

as I said, only for winevenlogs, you could filter events on Universal Forwarders, for more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Inputsconf

in few words, you have to add to the inputs.conf stanza:

blacklist = (?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1

anyway, are you sure that there isn't any Heavy Forwarder between Universal Forwarder and Indexers?

Then try to use an easier regex like:

ConfigSync\.ps1

you can test it in regex101.com or in a Splunk search to check if the regex checks your events (using the regex command):

index=your_index source=XmlWinEventLog:Microsoft-Windows-PowerShell/Operational | regex "(?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1"

Ciao.

Giuseppe

Re: Route and filter data

PickleRick — Wed, 18 May 2022 06:49:26 GMT

It's probably due to the order of precedence of stanzas and the fact that the events incoming from the UF do not have the source set as you want it.

The source file is being rewritten using transforms within the TA_windows app and you're trying to match the _resulting_ source. It won't match this way since splunk decides before applying transforms which stanzas are relevant for given event (as a result you can't - for example - create a loop with source rewrites; I tried :D)