Is it possible to filter specific field values in indexers without HeavyForwarder in indexer cluster ?

[khanlarloo]

Hi

Is it possible to filter specific field values in indexers without HeavyForwarder in indexer cluster ? 

[khanlarloo]

Thank You for answering my question, Then How should I configure the output.conf?

we have 60 indexer, and 5 Sh, and 500 UF with CM

[PickleRick]

Ok, let me rephrase the @gcusello 's answer a little.

Routing and filtering is something that is a part of index-time event processing and it happens on the first "heavy" (non-UF) splunk component in event's path. So if you send straight from UF's to indexers, it's gonna happen on indexers. If you have HF's in front of indexers you'd do it on HF's but if you don't, do it on indexers because they are the first "full" or "heavy" component.

What you should do is create a props and transforms settings according to https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad pack it into an app and push to your indexers as you'd do with any other app.

[gcusello]

Hi @khanlarloo,

outputs.conf isn't useful in events filtering, but it's useful only to configure your UFs to send logs to Indexers.

To filter events, you have to configure props.conf and transforms.conf on you Indexers, as described in the above links.

Ciao.

Giuseppe

[khanlarloo]

It doesn't work.

my props.conf

[XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
TRANSFORMS-null= ConfigSync-win

transforms.conf

[ConfigSync-win]
REGEX = Path=\"C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\"
DEST_KEY = queue
FORMAT = nullQueue

[gcusello]

Hi @khanlarloo,

are you sure that "XmlWinEventLog:Microsoft-Windows-PowerShell/Operational" is the sourcetype of your flow?

did you checked the regex  in transforms.conf?

Then, where do you located props.conf and transforms.conf?

they must be located on Indexers or (when present) on Heavy Forwarders.

Ciao.

Giuseppe

[khanlarloo]

"XmlWinEventLog:Microsoft-Windows-PowerShell/Operational"  is source,  ilocated transforms.conf  and props.con in indexeres under /$SPLUNK_HOME$/etc/slave-apps/_cluster/local

[gcusello]

Hi @khanlarloo,

in this case, the room name must be:

[source::XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]

if you don't use source or host, the name is always the sourcetype.

Ciao.

Giuseppe

[khanlarloo]

my raw log is :

</Data><Data Name='ScriptBlockId'>dd493fb0-9281-4197-a347-a7afaeea3632</Data><Data Name='Path'>C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1</Data></EventData></Event>

and i want to filter this part "C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1"

can you tell me what is the regex for it?

[PickleRick]

Wait, wait, wait...

Your raw data is like this:

<Data Name='Path'>C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1</Data>

and your regex is

Path=\"C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\"

 You don't have string fitting that regex in your event. You're trying to do some form of field value matching which will not work instead of regex match.

Try something like

'Path'>C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync.ps1<

 As @gcusello said - use https://regex101.com to verify your regex against your data

[khanlarloo]

i do it, but i still recieve this log.

transforms.conf

[setnull]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = (?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1

props.conf

[source::XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
TRANSFORMS-null= setnull

[PickleRick]

It's probably due to the order of precedence of stanzas and the fact that the events incoming from the UF do not have the source set as you want it.

The source file is being rewritten using transforms within the TA_windows app and you're trying to match the _resulting_ source. It won't match this way since splunk decides before applying transforms which stanzas are relevant for given event (as a result you can't - for example - create a loop with source rewrites; I tried :D)

[gcusello]

Hi @khanlarloo,

as I said, only for winevenlogs, you could filter events on Universal Forwarders, for more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Inputsconf

in few words, you have to add to the inputs.conf stanza:

blacklist = (?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1

anyway, are you sure that there isn't any Heavy Forwarder between Universal Forwarder and Indexers?

Then try to use an easier regex like:

ConfigSync\.ps1

you can test it in regex101.com or in a Splunk search to check if the regex checks your events (using the regex command):

index=your_index source=XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
| regex "(?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1"

Ciao.

Giuseppe

[gcusello]

Hi @khanlarloo,

this is your regex 

C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\<

that you can test at https://regex101.com/r/loRC2Y/1

Ciao.

Giuseppe

[gcusello]

Hi @khanlarloo,

filtering is possible only on Indexers (also clustered) or (when present) on Heavy Forwarders, it isn't possible (except Wineventlogs) on Universal Forwarders. you have to filter data on the first full Splunk instance the data encounter

In other words: if you have HFs in your architecture you have to filter data on HFs, otherwise on Indexers, it isn't possible to filer data on Indexers when you also have HFs.

But you want to filter the full events or a part of them?

If full events, you can see how to do it at https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad

if a part of them, you can see  how to do it at https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Anonymizedata

Ciao.

Giuseppe