topic Re: Configuration for a full forwarder - not filtering in Getting Data In

Configuration for a full forwarder - not filtering

dp546987 — Fri, 21 Mar 2014 16:43:15 GMT

Hi,
I've a full forwarder on machine A pointing at some log files in c:\temp*.log. These are being forwared to the full splunk install on machine B. I only want send the lines in the log files on machine A that contain the string [1:] or a [2:] to the splunk indexer from the forwarder on Machine B.

My \etc\apps\search\local\inputs.conf file looks like:

[monitor://c:\temp\log\*.log]
disabled = false

My \etc\apps\search\local\props.conf file looks like:

[splunkd]
EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<message>.+)

[splunk_web_service]
EXTRACT-useragent = userAgent=(?P<browser>[^ (]+)

[source:c:\temp\log\...]
TRANSFORMS-dp = setTypeOne, setTypeTwo

My \etc\apps\search\local\transforms.conf looks like:

#   Version 6.0.2
[setTypeOne]
DEST_KEY = MetaData:Sourcetype
REGEX = \[1:1\]
FORMAT = sourcetype::dp

[setTypeTwo]
DEST_KEY = MetaData:Sourcetype
REGEX = \[1:2\]
FORMAT = sourcetype::dp

The problem are:

No lines are being filtered out - the web app is showing all the lines in the file
The sourcetype dp is not being created

Any pointers would be gratefully received.
Thanks in advance.

Re: Configuration for a full forwarder - not filtering

kristian_kolb — Sat, 22 Mar 2014 09:36:22 GMT

Maybe I seriously misunderstanding something, but these configs do not do anything of the sort.

On your full forwarder, do the following assuming that you keep events that contain [1:] or [2:];

keep your inputs.conf as it is.

props.conf

[source::c:\temp\log\*.log]
TRANSFORMS-do_stuff = setOneTwo, setnull, keepOneTwo

transforms.conf

[setOneTwo]
DEST_KEY = MetaData:Sourcetype
REGEX = \[(1|2):\]
FORMAT = sourcetype::dp

[setnull]
DEST_KEY = queue
REGEX = .
FORMAT = nullQueue

[keepOneTwo]
DEST_KEY = queue
REGEX = \[(1|2):\]
FORMAT = indexQueue

The field extractions and that stuff you do one the search head/indexer.

EDIT: updated the source stanza in props.conf to reflect on the actual path (included wildcard). Should work better now.

Hope this helps,

Re: Configuration for a full forwarder - not filtering

dp546987 — Mon, 24 Mar 2014 09:28:49 GMT

Thanks, but this hasn't changed the behaviour at all. All the log file lines are still being posted, so no filtering is occuring at the heavy forwarder. The sourcetype is the name of the file, rather than dp.

Re: Configuration for a full forwarder - not filtering

dp546987 — Mon, 24 Mar 2014 09:31:50 GMT

Am I correct in where the config files are kept - \Splunk\etc\apps\search\local. I notice that there is a \Splunk\etc\apps\SplunkForwarder\local directory. What( if anything ) should be kept in there with respect to my aim of filtering the data sent to the remote splunk server.
Thanks

Re: Configuration for a full forwarder - not filtering

kristian_kolb — Mon, 24 Mar 2014 19:40:48 GMT

As long as the search app is enabled on the forwarder, you can put your settings there. However, to be absolutely sure that they are being honoured, you should put them in $SPLUNK_HOME/etc/system/local, where $SPLUNK_HOME the installation directory - normally c:\program files\splunk or /opt/splunk. This location overrides any other setting, but has the disadvantage that configurations here cannot be altered from a Deployment Server. However, I don't think config file precedence plays a part in the behaviour you're (not) seeing.

Re: Configuration for a full forwarder - not filtering

kristian_kolb — Mon, 24 Mar 2014 19:50:33 GMT

see update to answer above.