Hi All,
I have an issue where I need to cull certain IP sources and destinations from syslog sources before it gets indexed as these fill up our quota and we don't need to see it. I'm pretty sure I have the regex right even through its probably a but long winded but I've never used regex before.
What I need to do is put this in the transforms.conf file on the indexer but I'm not sure on the context I need to have so Splunk recognises it and culls it.
I have copied in what I have below, which isn't working, props file first then transforms. Its a Light Forwarder running in a deployment server environment
/opt/splunkforwarder/etc/system/local# cat props.conf
[loadbalancer]
NO_BINARY_CHECK = 1
pulldown_type = 1
TRANSFORMS-null = loadbalancer_setnull
/opt/splunkforwarder/etc/system/local# cat transforms.conf
[loadbalancer_setnull]
REGEX = dest_ip=\"172\.16\.100\.(6[5-9]|[7-8][0-9]|9[0-5])|224\.0\.0\.(5|18)|192\.168\.18[6-7]\.(0|16|48|240)\".*source_ip=\"172\.16\.\d+.\d+|0\.0\.0\.0\"
DEST_KEY = queue
FORMAT = nullQueue
there are a fair few combos we need to cull but 2 examples are:
dest 224.0.0.18 src 172.16.189.105
dest 192.168.187.48 src 0.0.0.0
Have been trying to get it work in the search app by prefixing with |rex field=_raw or |rex field=dest_ip and various combos but cannot get that working either.
Just coming back to add a bit of info for anyone who may have come across this themselves. I have now got this working after working on it off and on when I had spare time.
REGEX = dest_ip=(\"172\.16\.(148|250|181|190)\..*\"|\"10\.10\.\d+.\d+\"|\"224\.0\.0\.(5|18)\"|\"192\.168\.(196|197)\.(0|16|48|240)\").*source_ip=(\"172\.26\.\d+.\d+\"|\"0\.0\.0\.0\")
Hope this helps someone in my similar situation
How can i exclude entries with src=0.0.0.0 ?
Just coming back to add a bit of info for anyone who may have come across this themselves. I have now got this working after working on it off and on when I had spare time.
REGEX = dest_ip=(\"172\.16\.(148|250|181|190)\..*\"|\"10\.10\.\d+.\d+\"|\"224\.0\.0\.(5|18)\"|\"192\.168\.(196|197)\.(0|16|48|240)\").*source_ip=(\"172\.26\.\d+.\d+\"|\"0\.0\.0\.0\")
Hope this helps someone in my similar situation
If you want to discard certain events, the regex you set in transforms.conf does not need any capturing groups. It simply looks at the event and if it finds the regex, the event goes to the place you define (in your case nullQueue). Therefore, if you want to exclude ips with "dest"-values in your local network from 192.168.0.x to 192.168.2.x, you would set
REGEX = dest 192\.168\.[0-2]\.\d{1,3}
Now, depending on the combinations you want to exclude, these expressions can get quite complicated, but this should get you started. Perhaps as an idea to work with, it is sometimes easier to exclude a whole lot of events and then re-add them to the indexing queue. Your props.conf would then look something like this:
TRANSFORMS-set= setnull,setparsing
With setnull being the stanza in transforms.conf that sends events to nullqueue and setparsing another stanza which sends events to indexQueue instead of nullQueue.
Thanks very much jeffland this has pointed me in the right direction I have also since found out I need to move this off the forwarder to the indexer as the forwarder wont do this type of search.
Just as an aside, I can't seem to locate the transforms and props folders in the /home/splunk/idx01-configurations/etc/system/local# directory on the Indexer. Do I need to create these from scratch?
Awesome, btool b just what I needed. I have no idea which files are being used on the indexer, theres over 20 transform and props files that turned up in my search results and none were where I expected them to be so I will need to create these myself in the right spot. Thanks very much for your help.
You're welcome 🙂