Getting Data In

Is it possible to filter specific field values in indexers without HeavyForwarder in indexer cluster ?

khanlarloo
Explorer

Hi

Is it possible to filter specific field values in indexers without HeavyForwarder in indexer cluster 

 

Labels (1)
0 Karma

khanlarloo
Explorer

Thank You for answering my question, Then How should I configure the output.conf?

we have 60 indexer, and 5 Sh, and 500 UF with CM

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Ok, let me rephrase the @gcusello 's answer a little.

Routing and filtering is something that is a part of index-time event processing and it happens on the first "heavy" (non-UF) splunk component in event's path. So if you send straight from UF's to indexers, it's gonna happen on indexers. If you have HF's in front of indexers you'd do it on HF's but if you don't, do it on indexers because they are the first "full" or "heavy" component.

What you should do is create a props and transforms settings according to https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad pack it into an app and push to your indexers as you'd do with any other app.

 

gcusello
SplunkTrust
SplunkTrust

Hi @khanlarloo,

outputs.conf isn't useful in events filtering, but it's useful only to configure your UFs to send logs to Indexers.

To filter events, you have to configure props.conf and transforms.conf on you Indexers, as described in the above links.

Ciao.

Giuseppe

 

0 Karma

khanlarloo
Explorer

It doesn't work.

my props.conf

[XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
TRANSFORMS-null= ConfigSync-win

transforms.conf

[ConfigSync-win]
REGEX = Path=\"C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\"
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @khanlarloo,

are you sure that "XmlWinEventLog:Microsoft-Windows-PowerShell/Operational" is the sourcetype of your flow?

did you checked the regex  in transforms.conf?

Then, where do you located props.conf and transforms.conf?

they must be located on Indexers or (when present) on Heavy Forwarders.

Ciao.

Giuseppe

0 Karma

khanlarloo
Explorer

"XmlWinEventLog:Microsoft-Windows-PowerShell/Operational"  is source,  ilocated transforms.conf  and props.con in indexeres under /$SPLUNK_HOME$/etc/slave-apps/_cluster/local

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @khanlarloo,

in this case, the room name must be:

[source::XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]

if you don't use source or host, the name is always the sourcetype.

Ciao.

Giuseppe

0 Karma

khanlarloo
Explorer

my raw log is :

</Data><Data Name='ScriptBlockId'>dd493fb0-9281-4197-a347-a7afaeea3632</Data><Data Name='Path'>C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1</Data></EventData></Event>

and i want to filter this part "C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1"

can you tell me what is the regex for it?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Wait, wait, wait...

Your raw data is like this:

<Data Name='Path'>C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1</Data>

and your regex is

Path=\"C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\"

 You don't have string fitting that regex in your event. You're trying to do some form of field value matching which will not work instead of regex match.

Try something like

'Path'>C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync.ps1<

 As @gcusello said - use https://regex101.com to verify your regex against your data

0 Karma

khanlarloo
Explorer

i do it, but i still recieve this log.

transforms.conf

[setnull]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = (?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1

props.conf

[source::XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
TRANSFORMS-null= setnull

Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

It's probably due to the order of precedence of stanzas and the fact that the events incoming from the UF do not have the source set as you want it.

The source file is being rewritten using transforms within the TA_windows app and you're trying to match the _resulting_ source. It won't match this way since splunk decides before applying transforms which stanzas are relevant for given event (as a result you can't - for example - create a loop with source rewrites; I tried :D)

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @khanlarloo,

as I said, only for winevenlogs, you could filter events on Universal Forwarders, for more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Inputsconf

in few words, you have to add to the inputs.conf stanza:

blacklist = (?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1

anyway, are you sure that there isn't any Heavy Forwarder between Universal Forwarder and Indexers?

Then try to use an easier regex like:

ConfigSync\.ps1

you can test it in regex101.com or in a Splunk search to check if the regex checks your events (using the regex command):

index=your_index source=XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
| regex "(?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1"

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @khanlarloo,

this is your regex 

C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\<

that you can test at https://regex101.com/r/loRC2Y/1

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @khanlarloo,

filtering is possible only on Indexers (also clustered) or (when present) on Heavy Forwarders, it isn't possible (except Wineventlogs) on Universal Forwarders. you have to filter data on the first full Splunk instance the data encounter

In other words: if you have HFs in your architecture you have to filter data on HFs, otherwise on Indexers, it isn't possible to filer data on Indexers when you also have HFs.

But you want to filter the full events or a part of them?

If full events, you can see how to do it at https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad

if a part of them, you can see  how to do it at https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Anonymizedata

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...