Getting Data In

Drop events before indexing. Complex filter

hcpr
Path Finder

Hi,
I need to filter out some data before indexing, ands can't quite get it to work.
The data is a json format (from Suricata), and looks like this typically:

{"timestamp":"2017-01-09T12:24:38.870864+0100","flow_id":1556450832173294,"in_iface":"sniffer0","event_type":"dns","vlan":935,"src_ip":"192.168.7.52","src_port":53,"dest_ip":"192.168.7.27","dest_port":52670,"proto":"UDP","dns":{"type":"answer","id":27371,"rcode":"NOERROR","rrname":"253.165.168.192.in-addr.arpa","rrtype":"PTR","ttl":11476,"rdata":"pri-f5-indre.proxy.u.dep.no"}}
{"timestamp":"2017-01-09T12:24:38.871589+0100","flow_id":1088479785536677,"in_iface":"sniffer0","event_type":"dns","vlan":190,"src_ip":"192.168.149.30","src_port":35577,"dest_ip":"192.168.7.52","dest_port":53,"proto":"UDP","dns":{"type":"query","id":53493,"rrname":"cdn.els-cdn.com","rrtype":"A","tx_id":0}}

(that is two long lines)
This is the dns log from Suricata written to a json file.
What I need to do is first filter so that I only get the traffic to/from a few servere (some dns servers), after that I need to filter out all PTR logs.
The first part was easy. Since the machine in question is running a heavy forwarder I added the following to the config:

In props.conf:

[source::/data/suricata/log/eve-dns.json]
TRANSFORMS-set = setnull,setparsing

And in transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (?:(?:"vlan":906.*?"dest_ip":"192\.168\.7\.5[2345]","dest_port":53)|(?:"vlan":710.*?"dest_ip":"192\.168\.4\.[45678]","dest_port":53)|(?:"vlan":850.*?"dest_ip":"192\.168\.156\.[234]","dest_port":53)|(?:"vlan":370.*?"dest_ip":"192\.168\.137\.[23]","dest_port":53)|(?:"vlan":311.*?"dest_ip":"192\.168\.252\.[56]","dest_port":53))|(?:(?:"vlan":906.*?"src_ip":"192\.168\.7\.5[2345]","src_port":53)|(?:"vlan":710.*?"src_ip":"192\.168\.4\.[45678]","src_port":53)|(?:"vlan":850.*?"src_ip":"192\.168\.156\.[234]","src_port":53)|(?:"vlan":370.*?"src_ip":"192\.168\.137\.[23]","src_port":53)|(?:"vlan":311.*?"src_ip":"192\.168\.252\.[56]","src_port":53))
DEST_KEY = queue
FORMAT = indexQueue

But I can't figure out. First I tried adding a third transform to the mix, on the theory that they are run in order, and that the forst one stops. That does not seem to be tha case, as the the following did not fork:

In props.conf:

[source::/data/suricata/log/eve-dns.json]
TRANSFORMS-set = setnull,setparsing,setfilter

And adding the following in transforms.conf

 [setfilter]
REGEX = "rrtype":"PTR"
DST_KEY = queue
FORMAT = nullQueue   

I've also tried using SEDCMD, which I've had success with in other problems. For instance this (and several variations of the regexp):

SEDCMD-remove_PTR = s/*."rrtype":"PTR".*//g

So, any ideas anyone?

0 Karma
1 Solution

hcpr
Path Finder

Just for reference, in case someone else makes the same mistake.
As pointed out by alemarzu, I had a typo in one of the stanzas.
Also, if you are doing this, be aware that the documentation of props.conf is wrong. The last element in the TRANSFORMS-= line is done first, then the next and so on. Not the other way as the documentation says.
Took me several tests to confirm.

View solution in original post

0 Karma

hcpr
Path Finder

Just for reference, in case someone else makes the same mistake.
As pointed out by alemarzu, I had a typo in one of the stanzas.
Also, if you are doing this, be aware that the documentation of props.conf is wrong. The last element in the TRANSFORMS-= line is done first, then the next and so on. Not the other way as the documentation says.
Took me several tests to confirm.

0 Karma

somesoni2
Revered Legend

I would not say the documentation is wrong but may be incomplete. The event routing (transforms with DEST_KEY=queue) are special type of transform and the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue. See more details here:
http://docs.splunk.com/Documentation/SplunkCloud/6.5.1/Forwarding/Routeandfilterdatad#Keep_specific_...

0 Karma

alemarzu
Motivator

Hi @hcpr

You have a typo on your setfilter stanza, its DEST_KEY not DST_KEY. Also, you should be scaping doble quotes on your regexes.

Hope it helps.

0 Karma

hcpr
Path Finder

You are absolutely right. DST_KEY was one of the problems.

0 Karma

alemarzu
Motivator

Glad it worked out, happy splunking!

0 Karma

gcusello
Legend

Hi hcpr,
setnull must be always the first, but did you try to change the order between setparsing and setfilter?
Bye.
Giuseppe

0 Karma

hcpr
Path Finder

Yes, I've tried that. Same result though.

0 Karma

hcpr
Path Finder

Actually, this was relevant but hidden by the typo I had in one of the transforms stanzas.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...