I'm wanting to exclude records with a particular keyword from being ingested by the indexer.
I have several Windows servers all pointing to a heavy forwarder where the inputs.conf file determines which logs to ingest into the Splunk indexer however there is some selected content that I want to exclude that exists in some of the included logs.
Specifically, I want to exclude any records that contain the word "Zabbix", or "Zabbix Agent".
How can this be done and where is the best place to do this filtering?
Hi dyude @balcv ,
You can write a props and transforms for this ...
props.conf
[Your sourcetype]
TRANSFORMS-set= zabbix,zabbix_agent
transforms.conf
[zabbix]
REGEX = Zabbix
DEST_KEY = queue
FORMAT = nullQueue
[zabbix_agent]
REGEX = Zabbix\sAgent
DEST_KEY = queue
FORMAT = nullQueue
This will exclude all the Zabbix
and Zabbix Agent
keywords present in the logs.
Try this out and let me know if it works for you!
Hi dyude @balcv ,
You can write a props and transforms for this ...
props.conf
[Your sourcetype]
TRANSFORMS-set= zabbix,zabbix_agent
transforms.conf
[zabbix]
REGEX = Zabbix
DEST_KEY = queue
FORMAT = nullQueue
[zabbix_agent]
REGEX = Zabbix\sAgent
DEST_KEY = queue
FORMAT = nullQueue
This will exclude all the Zabbix
and Zabbix Agent
keywords present in the logs.
Try this out and let me know if it works for you!
It looks like the config details provided by vinod94 were in fact correct however I needed to modify the props.conf and transforms.conf on the indexer box and NOT on the heavy forwarder.
When I worked through the data flow, the heavy forwarder is only being used as the deployment server and not receiving the logs for these specific data sources. Once I updated the files on the indexer, I got the exact results I was hoping for.
Thank you.
Glad! it worked for you! (Y)
Thanks for the details @vindod94 . Much appreciated.
One question, in the props.conf, you have [Your sourcetype]. What should be in this header? Does it relate to a windows log or is it just a name I assign it?
Thanks
@balcv,
You just have to put Your Sourcetype Name
for which you are filtering the logs . So basically you can do this for a host OR source OR sourcetype.
You can follow this props.conf doc, this will give you an idea.
https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/Propsconf
Thanks very much. I have added the code as suggested, and restarted the heavy forwarder, however the Zabbix items are still getting through to the indexer.
@balcv,
Have you applied it on your sourcetype(your sourcetype name)?
I think so, yes.
Props.conf
[source::WinEventLog:Application]
TRANSFORMS-set= zabbix
Data according to indexer:
index="winEventLog"
3/22/19 8:22:52.000 AM 03/22/2019
08:22:52 AM LogName=Application
SourceName=Zabbix Agent EventCode=1
EventType=3 Show all 19 lines host
= EXIGE source = WinEventLog:Application
Does this look correct?