Getting Data In

Drop events before indexing. Complex filter

Path Finder

Hi,
I need to filter out some data before indexing, ands can't quite get it to work.
The data is a json format (from Suricata), and looks like this typically:

{"timestamp":"2017-01-09T12:24:38.870864+0100","flow_id":1556450832173294,"in_iface":"sniffer0","event_type":"dns","vlan":935,"src_ip":"192.168.7.52","src_port":53,"dest_ip":"192.168.7.27","dest_port":52670,"proto":"UDP","dns":{"type":"answer","id":27371,"rcode":"NOERROR","rrname":"253.165.168.192.in-addr.arpa","rrtype":"PTR","ttl":11476,"rdata":"pri-f5-indre.proxy.u.dep.no"}}
{"timestamp":"2017-01-09T12:24:38.871589+0100","flow_id":1088479785536677,"in_iface":"sniffer0","event_type":"dns","vlan":190,"src_ip":"192.168.149.30","src_port":35577,"dest_ip":"192.168.7.52","dest_port":53,"proto":"UDP","dns":{"type":"query","id":53493,"rrname":"cdn.els-cdn.com","rrtype":"A","tx_id":0}}

(that is two long lines)
This is the dns log from Suricata written to a json file.
What I need to do is first filter so that I only get the traffic to/from a few servere (some dns servers), after that I need to filter out all PTR logs.
The first part was easy. Since the machine in question is running a heavy forwarder I added the following to the config:

In props.conf:

[source::/data/suricata/log/eve-dns.json]
TRANSFORMS-set = setnull,setparsing

And in transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (?:(?:"vlan":906.*?"dest_ip":"192\.168\.7\.5[2345]","dest_port":53)|(?:"vlan":710.*?"dest_ip":"192\.168\.4\.[45678]","dest_port":53)|(?:"vlan":850.*?"dest_ip":"192\.168\.156\.[234]","dest_port":53)|(?:"vlan":370.*?"dest_ip":"192\.168\.137\.[23]","dest_port":53)|(?:"vlan":311.*?"dest_ip":"192\.168\.252\.[56]","dest_port":53))|(?:(?:"vlan":906.*?"src_ip":"192\.168\.7\.5[2345]","src_port":53)|(?:"vlan":710.*?"src_ip":"192\.168\.4\.[45678]","src_port":53)|(?:"vlan":850.*?"src_ip":"192\.168\.156\.[234]","src_port":53)|(?:"vlan":370.*?"src_ip":"192\.168\.137\.[23]","src_port":53)|(?:"vlan":311.*?"src_ip":"192\.168\.252\.[56]","src_port":53))
DEST_KEY = queue
FORMAT = indexQueue

But I can't figure out. First I tried adding a third transform to the mix, on the theory that they are run in order, and that the forst one stops. That does not seem to be tha case, as the the following did not fork:

In props.conf:

[source::/data/suricata/log/eve-dns.json]
TRANSFORMS-set = setnull,setparsing,setfilter

And adding the following in transforms.conf

 [setfilter]
REGEX = "rrtype":"PTR"
DST_KEY = queue
FORMAT = nullQueue   

I've also tried using SEDCMD, which I've had success with in other problems. For instance this (and several variations of the regexp):

SEDCMD-remove_PTR = s/*."rrtype":"PTR".*//g

So, any ideas anyone?

0 Karma
1 Solution

Path Finder

Just for reference, in case someone else makes the same mistake.
As pointed out by alemarzu, I had a typo in one of the stanzas.
Also, if you are doing this, be aware that the documentation of props.conf is wrong. The last element in the TRANSFORMS-= line is done first, then the next and so on. Not the other way as the documentation says.
Took me several tests to confirm.

View solution in original post

0 Karma

Path Finder

Just for reference, in case someone else makes the same mistake.
As pointed out by alemarzu, I had a typo in one of the stanzas.
Also, if you are doing this, be aware that the documentation of props.conf is wrong. The last element in the TRANSFORMS-= line is done first, then the next and so on. Not the other way as the documentation says.
Took me several tests to confirm.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

I would not say the documentation is wrong but may be incomplete. The event routing (transforms with DESTKEY=queue) are special type of transform and the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue. See more details here:
http://docs.splunk.com/Documentation/SplunkCloud/6.5.1/Forwarding/Routeandfilterdatad#Keep
specificeventsanddiscardthe_rest

0 Karma

Motivator

Hi @hcpr

You have a typo on your setfilter stanza, its DESTKEY not DSTKEY. Also, you should be scaping doble quotes on your regexes.

Hope it helps.

0 Karma

Path Finder

You are absolutely right. DST_KEY was one of the problems.

0 Karma

Motivator

Glad it worked out, happy splunking!

0 Karma

Legend

Hi hcpr,
setnull must be always the first, but did you try to change the order between setparsing and setfilter?
Bye.
Giuseppe

0 Karma

Path Finder

Yes, I've tried that. Same result though.

0 Karma

Path Finder

Actually, this was relevant but hidden by the typo I had in one of the transforms stanzas.

0 Karma