Hi,
I need to filter out some data before indexing, ands can't quite get it to work.
The data is a json format (from Suricata), and looks like this typically:
{"timestamp":"2017-01-09T12:24:38.870864+0100","flow_id":1556450832173294,"in_iface":"sniffer0","event_type":"dns","vlan":935,"src_ip":"192.168.7.52","src_port":53,"dest_ip":"192.168.7.27","dest_port":52670,"proto":"UDP","dns":{"type":"answer","id":27371,"rcode":"NOERROR","rrname":"253.165.168.192.in-addr.arpa","rrtype":"PTR","ttl":11476,"rdata":"pri-f5-indre.proxy.u.dep.no"}}
{"timestamp":"2017-01-09T12:24:38.871589+0100","flow_id":1088479785536677,"in_iface":"sniffer0","event_type":"dns","vlan":190,"src_ip":"192.168.149.30","src_port":35577,"dest_ip":"192.168.7.52","dest_port":53,"proto":"UDP","dns":{"type":"query","id":53493,"rrname":"cdn.els-cdn.com","rrtype":"A","tx_id":0}}
(that is two long lines)
This is the dns log from Suricata written to a json file.
What I need to do is first filter so that I only get the traffic to/from a few servere (some dns servers), after that I need to filter out all PTR logs.
The first part was easy. Since the machine in question is running a heavy forwarder I added the following to the config:
In props.conf:
[source::/data/suricata/log/eve-dns.json]
TRANSFORMS-set = setnull,setparsing
And in transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = (?:(?:"vlan":906.*?"dest_ip":"192\.168\.7\.5[2345]","dest_port":53)|(?:"vlan":710.*?"dest_ip":"192\.168\.4\.[45678]","dest_port":53)|(?:"vlan":850.*?"dest_ip":"192\.168\.156\.[234]","dest_port":53)|(?:"vlan":370.*?"dest_ip":"192\.168\.137\.[23]","dest_port":53)|(?:"vlan":311.*?"dest_ip":"192\.168\.252\.[56]","dest_port":53))|(?:(?:"vlan":906.*?"src_ip":"192\.168\.7\.5[2345]","src_port":53)|(?:"vlan":710.*?"src_ip":"192\.168\.4\.[45678]","src_port":53)|(?:"vlan":850.*?"src_ip":"192\.168\.156\.[234]","src_port":53)|(?:"vlan":370.*?"src_ip":"192\.168\.137\.[23]","src_port":53)|(?:"vlan":311.*?"src_ip":"192\.168\.252\.[56]","src_port":53))
DEST_KEY = queue
FORMAT = indexQueue
But I can't figure out. First I tried adding a third transform to the mix, on the theory that they are run in order, and that the forst one stops. That does not seem to be tha case, as the the following did not fork:
In props.conf:
[source::/data/suricata/log/eve-dns.json]
TRANSFORMS-set = setnull,setparsing,setfilter
And adding the following in transforms.conf
[setfilter]
REGEX = "rrtype":"PTR"
DST_KEY = queue
FORMAT = nullQueue
I've also tried using SEDCMD, which I've had success with in other problems. For instance this (and several variations of the regexp):
SEDCMD-remove_PTR = s/*."rrtype":"PTR".*//g
So, any ideas anyone?
Just for reference, in case someone else makes the same mistake.
As pointed out by alemarzu, I had a typo in one of the stanzas.
Also, if you are doing this, be aware that the documentation of props.conf is wrong. The last element in the TRANSFORMS-= line is done first, then the next and so on. Not the other way as the documentation says.
Took me several tests to confirm.
Just for reference, in case someone else makes the same mistake.
As pointed out by alemarzu, I had a typo in one of the stanzas.
Also, if you are doing this, be aware that the documentation of props.conf is wrong. The last element in the TRANSFORMS-= line is done first, then the next and so on. Not the other way as the documentation says.
Took me several tests to confirm.
I would not say the documentation is wrong but may be incomplete. The event routing (transforms with DEST_KEY=queue) are special type of transform and the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue. See more details here:
http://docs.splunk.com/Documentation/SplunkCloud/6.5.1/Forwarding/Routeandfilterdatad#Keep_specific_...
Hi @hcpr
You have a typo on your setfilter stanza, its DEST_KEY not DST_KEY. Also, you should be scaping doble quotes on your regexes.
Hope it helps.
You are absolutely right. DST_KEY was one of the problems.
Glad it worked out, happy splunking!
Hi hcpr,
setnull must be always the first, but did you try to change the order between setparsing and setfilter?
Bye.
Giuseppe
Yes, I've tried that. Same result though.
Actually, this was relevant but hidden by the typo I had in one of the transforms stanzas.