topic Re: Monitor,transforms? in Monitoring Splunk

Monitor,transforms?

jgauthier — Wed, 23 Mar 2011 09:19:56 GMT

I am trying to monitor a file on a forwarder. I don't wish to send all the contents, as there is unnecessary data in it.

Please tell me if I am doing this correctly, because everything is being logged, not just things that match "TCP"

props.conf

[source::///var/log/remote/remotefirewall]
TRANSFORMS-routing=remoteASA

inputs.conf

[monitor:///var/log/remote/remotefirewall]

transforms.conf

[remoteASA]
REGEX = (?msi)TCP

Basic setup for testing. Once I get it working, I have a more advanced regex I need to apply.

Thanks

Re: Monitor,transforms?

dwaddle — Wed, 23 Mar 2011 20:11:42 GMT

The best practice way to only index some events is by using the nullQueue functionality. See http://www.splunk.com/base/Documentation/4.2/Deploy/Routeandfilterdatad for information on how to configure nullQueue. There are also several answers questions on it as well.

A light forwarder or universal forwarder will always send the whole content of the file to the indexer - as no parsing is done by these forwarders. A full (or heavy) forwarder will parse the events locally at the forwarder.

If you are using a light or universal forwarder, the nullQueue configuration must be done at the indexer. All of the data from the file will be sent to the indexer first and filtered there.

If you a using a heavy forwarder, the nullQueue configuration must be done on that forwarder - and it will perform local filtering before passing on preparsed events.

Typical practice today is the light forwarder (or universal in 4.2) approach, unless you have specific reasons otherwise. It does use extra bandwidth, however.

Re: Monitor,transforms?

jgauthier — Wed, 23 Mar 2011 23:30:38 GMT

Thanks. That was really helpful. I actually had that document open and was reading it. What I hadn't reached was the part about null, and discarding.

I am going to retry this.
I am interested in the light-forwarders. Understanding they don't filter, but the indexer will. I do have a follow up question. Let's assume I have three devices. Each with lots of logs. One does DHCP, one DNS, and the Last is a Wireless AP.

I want to send these all to a syslog receiver. Would I do something like this?

In props.conf:

[syslog]
TRANSFORMS-set= setnull,DNS,DHCP,Wireless

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[DNS]
REGEX = "insert regex to parse for DNS from device 'B'"
DEST_KEY = queue
FORMAT = indexQueue

[DHCP]
REGEX = insert regex to parse for DHCP from device 'A'
DEST_KEY = queue
FORMAT = indexQueue

[Wireless]
REGEX = .
DEST_KEY = insert regex to match wireless device IP
FORMAT = indexQueue

Re: Monitor,transforms?

jgauthier — Thu, 24 Mar 2011 04:23:37 GMT

This is driving me nuts. I am missing something fundamental.

[source::.../var/log/remote/remotefirewall]
sourcetype=cisco_asa
TRANSFORMS-set= remoteASA

[remoteASA]
REGEX = TCP
DEST_KEY = _TCP_ROUTING
FORMAT = tcpout

This is logging everything in it that does NOT contain "TCP". I thought it would log the other way. Shouldn't it log what matches TCP?

I change it to:
REGEX = (?i)deny
and it stops logging anything with [dD]eny.

This seems to completely negate what this says: http://www.splunk.com/base/Documentation/4.2/Deploy/Routeandfilterdatad

I got here because I was trying to follow the "Keep specific events and discard the rest" section, and I was having absolutely no luck.

so, I started to isolate the config.

Thanks for the help/explanation.

Re: Monitor,transforms?

jgauthier — Fri, 25 Mar 2011 06:24:53 GMT

I was able to solve this. Basically, on a forwarder you always send the output queue to indexQueue. I was manually setting it above to the forwarder config. This did not work.

Changing it to the indexQueue made all the magic happen.