Hi
Is it possible to filter specific field values in indexers without HeavyForwarder in indexer cluster ?
Thank You for answering my question, Then How should I configure the output.conf?
we have 60 indexer, and 5 Sh, and 500 UF with CM
Ok, let me rephrase the @gcusello 's answer a little.
Routing and filtering is something that is a part of index-time event processing and it happens on the first "heavy" (non-UF) splunk component in event's path. So if you send straight from UF's to indexers, it's gonna happen on indexers. If you have HF's in front of indexers you'd do it on HF's but if you don't, do it on indexers because they are the first "full" or "heavy" component.
What you should do is create a props and transforms settings according to https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad pack it into an app and push to your indexers as you'd do with any other app.
Hi @khanlarloo,
outputs.conf isn't useful in events filtering, but it's useful only to configure your UFs to send logs to Indexers.
To filter events, you have to configure props.conf and transforms.conf on you Indexers, as described in the above links.
Ciao.
Giuseppe
It doesn't work.
my props.conf
[XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
TRANSFORMS-null= ConfigSync-win
transforms.conf
[ConfigSync-win]
REGEX = Path=\"C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\"
DEST_KEY = queue
FORMAT = nullQueue
Hi @khanlarloo,
are you sure that "XmlWinEventLog:Microsoft-Windows-PowerShell/Operational" is the sourcetype of your flow?
did you checked the regex in transforms.conf?
Then, where do you located props.conf and transforms.conf?
they must be located on Indexers or (when present) on Heavy Forwarders.
Ciao.
Giuseppe
"XmlWinEventLog:Microsoft-Windows-PowerShell/Operational" is source, ilocated transforms.conf and props.con in indexeres under /$SPLUNK_HOME$/etc/slave-apps/_cluster/local
Hi @khanlarloo,
in this case, the room name must be:
[source::XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
if you don't use source or host, the name is always the sourcetype.
Ciao.
Giuseppe
my raw log is :
</Data><Data Name='ScriptBlockId'>dd493fb0-9281-4197-a347-a7afaeea3632</Data><Data Name='Path'>C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1</Data></EventData></Event>
and i want to filter this part "C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1"
can you tell me what is the regex for it?
Wait, wait, wait...
Your raw data is like this:
<Data Name='Path'>C:\Program Files\Citrix\ConfigSync\ConfigSync.ps1</Data>
and your regex is
Path=\"C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\"
You don't have string fitting that regex in your event. You're trying to do some form of field value matching which will not work instead of regex match.
Try something like
'Path'>C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync.ps1<
As @gcusello said - use https://regex101.com to verify your regex against your data
i do it, but i still recieve this log.
transforms.conf
[setnull]
DEST_KEY = queue
FORMAT = nullQueue
REGEX = (?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1
props.conf
[source::XmlWinEventLog:Microsoft-Windows-PowerShell/Operational]
TRANSFORMS-null= setnull
It's probably due to the order of precedence of stanzas and the fact that the events incoming from the UF do not have the source set as you want it.
The source file is being rewritten using transforms within the TA_windows app and you're trying to match the _resulting_ source. It won't match this way since splunk decides before applying transforms which stanzas are relevant for given event (as a result you can't - for example - create a loop with source rewrites; I tried :D)
Hi @khanlarloo,
as I said, only for winevenlogs, you could filter events on Universal Forwarders, for more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Inputsconf
in few words, you have to add to the inputs.conf stanza:
blacklist = (?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1
anyway, are you sure that there isn't any Heavy Forwarder between Universal Forwarder and Indexers?
Then try to use an easier regex like:
ConfigSync\.ps1
you can test it in regex101.com or in a Splunk search to check if the regex checks your events (using the regex command):
index=your_index source=XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
| regex "(?i)C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1"
Ciao.
Giuseppe
Hi @khanlarloo,
this is your regex
C:\\Program Files\\Citrix\\ConfigSync\\ConfigSync\.ps1\<
that you can test at https://regex101.com/r/loRC2Y/1
Ciao.
Giuseppe
Hi @khanlarloo,
filtering is possible only on Indexers (also clustered) or (when present) on Heavy Forwarders, it isn't possible (except Wineventlogs) on Universal Forwarders. you have to filter data on the first full Splunk instance the data encounter
In other words: if you have HFs in your architecture you have to filter data on HFs, otherwise on Indexers, it isn't possible to filer data on Indexers when you also have HFs.
But you want to filter the full events or a part of them?
If full events, you can see how to do it at https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad
if a part of them, you can see how to do it at https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Anonymizedata
Ciao.
Giuseppe