topic Re: Drop events before indexing. Complex filter in Getting Data In

Drop events before indexing. Complex filter

hcpr — Mon, 09 Jan 2017 12:15:54 GMT

Hi,
I need to filter out some data before indexing, ands can't quite get it to work.
The data is a json format (from Suricata), and looks like this typically:

{"timestamp":"2017-01-09T12:24:38.870864+0100","flow_id":1556450832173294,"in_iface":"sniffer0","event_type":"dns","vlan":935,"src_ip":"192.168.7.52","src_port":53,"dest_ip":"192.168.7.27","dest_port":52670,"proto":"UDP","dns":{"type":"answer","id":27371,"rcode":"NOERROR","rrname":"253.165.168.192.in-addr.arpa","rrtype":"PTR","ttl":11476,"rdata":"pri-f5-indre.proxy.u.dep.no"}}
{"timestamp":"2017-01-09T12:24:38.871589+0100","flow_id":1088479785536677,"in_iface":"sniffer0","event_type":"dns","vlan":190,"src_ip":"192.168.149.30","src_port":35577,"dest_ip":"192.168.7.52","dest_port":53,"proto":"UDP","dns":{"type":"query","id":53493,"rrname":"cdn.els-cdn.com","rrtype":"A","tx_id":0}}

(that is two long lines)
This is the dns log from Suricata written to a json file.
What I need to do is first filter so that I only get the traffic to/from a few servere (some dns servers), after that I need to filter out all PTR logs.
The first part was easy. Since the machine in question is running a heavy forwarder I added the following to the config:

In props.conf:

[source::/data/suricata/log/eve-dns.json]
TRANSFORMS-set = setnull,setparsing

And in transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (?:(?:"vlan":906.*?"dest_ip":"192\.168\.7\.5[2345]","dest_port":53)|(?:"vlan":710.*?"dest_ip":"192\.168\.4\.[45678]","dest_port":53)|(?:"vlan":850.*?"dest_ip":"192\.168\.156\.[234]","dest_port":53)|(?:"vlan":370.*?"dest_ip":"192\.168\.137\.[23]","dest_port":53)|(?:"vlan":311.*?"dest_ip":"192\.168\.252\.[56]","dest_port":53))|(?:(?:"vlan":906.*?"src_ip":"192\.168\.7\.5[2345]","src_port":53)|(?:"vlan":710.*?"src_ip":"192\.168\.4\.[45678]","src_port":53)|(?:"vlan":850.*?"src_ip":"192\.168\.156\.[234]","src_port":53)|(?:"vlan":370.*?"src_ip":"192\.168\.137\.[23]","src_port":53)|(?:"vlan":311.*?"src_ip":"192\.168\.252\.[56]","src_port":53))
DEST_KEY = queue
FORMAT = indexQueue

But I can't figure out. First I tried adding a third transform to the mix, on the theory that they are run in order, and that the forst one stops. That does not seem to be tha case, as the the following did not fork:

In props.conf:

[source::/data/suricata/log/eve-dns.json]
TRANSFORMS-set = setnull,setparsing,setfilter

And adding the following in transforms.conf

 [setfilter]
REGEX = "rrtype":"PTR"
DST_KEY = queue
FORMAT = nullQueue

I've also tried using SEDCMD, which I've had success with in other problems. For instance this (and several variations of the regexp):

SEDCMD-remove_PTR = s/*."rrtype":"PTR".*//g

So, any ideas anyone?

Re: Drop events before indexing. Complex filter

gcusello — Mon, 09 Jan 2017 13:16:24 GMT

Hi hcpr,
setnull must be always the first, but did you try to change the order between setparsing and setfilter?
Bye.
Giuseppe

Re: Drop events before indexing. Complex filter

hcpr — Mon, 09 Jan 2017 13:24:09 GMT

Yes, I've tried that. Same result though.

Re: Drop events before indexing. Complex filter

alemarzu — Tue, 29 Sep 2020 12:20:17 GMT

Hi @hcpr

You have a typo on your setfilter stanza, its DEST_KEY not DST_KEY. Also, you should be scaping doble quotes on your regexes.

Hope it helps.

Re: Drop events before indexing. Complex filter

hcpr — Tue, 10 Jan 2017 14:51:34 GMT

You are absolutely right. DST_KEY was one of the problems.

Re: Drop events before indexing. Complex filter

hcpr — Tue, 10 Jan 2017 14:54:29 GMT

Just for reference, in case someone else makes the same mistake.
As pointed out by alemarzu, I had a typo in one of the stanzas.
Also, if you are doing this, be aware that the documentation of props.conf is wrong. The last element in the TRANSFORMS-= line is done first, then the next and so on. Not the other way as the documentation says.
Took me several tests to confirm.

Re: Drop events before indexing. Complex filter

hcpr — Tue, 10 Jan 2017 14:55:21 GMT

Actually, this was relevant but hidden by the typo I had in one of the transforms stanzas.

Re: Drop events before indexing. Complex filter

somesoni2 — Tue, 29 Sep 2020 12:20:50 GMT

I would not say the documentation is wrong but may be incomplete. The event routing (transforms with DEST_KEY=queue) are special type of transform and the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue. See more details here:
http://docs.splunk.com/Documentation/SplunkCloud/6.5.1/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

Re: Drop events before indexing. Complex filter

alemarzu — Wed, 11 Jan 2017 14:56:18 GMT

Glad it worked out, happy splunking!