topic Re: Filtering specific log file on indexer in Getting Data In

Filtering specific log file on indexer

timmy13 — Sun, 26 Feb 2012 20:05:07 GMT

I have a very chatty forwarder that I do not have access to, so cannot stop the noise.

I have identified the log file that is sending millions of events an hour.

I would like to use props.conf and transforms.conf to filter this log out pre-index time, but it's not working like I would expect. The log file (NOV.log) is in a deep path, but I just want to use the name and no worry about the full path.

Here's my config:

props.conf
[source::...NOV.log}
TRANSFORMS-Filter_Events = FilterNOVlog

transforms.conf
[FilterNOVlog]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

Am I missing something really stupid, or shouldn't this work?

Thanks in advance!

Tim

Re: Filtering specific log file on indexer

jgedeon120 — Sun, 26 Feb 2012 21:27:36 GMT

Yes that should work, after you correct the } to a ]. Have you restarted Splunk?

Re: Filtering specific log file on indexer

timmy13 — Sun, 26 Feb 2012 23:24:51 GMT

Yah, I noticed that typo too, but it isn't working. ANd yes, I restarted splunk.

Re: Filtering specific log file on indexer

_d_ — Mon, 27 Feb 2012 00:29:59 GMT

Your props and transforms look correctly set for nullQueue-ing. Here are a couple of things you can try to troubleshoot the issue:

Is the source of the that data actually ...NOV.log? Perhaps it is being overwritten at inputs.conf at the forwarder.
Use btool to see the running configuration of Splunk for that source: ./splunk cmd btool props list my_source

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

Re: Filtering specific log file on indexer

timmy13 — Mon, 28 Sep 2020 11:26:36 GMT

Yes, the source is actually C:\Program Files\Common Files\xxx\xxx.xx.ProcessorService\xxx\logs\NOV.log, but I tried it with the full path as well and it didn't work.

Here's the btool output...
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
LEARN_SOURCETYPE = true
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800

Re: Filtering specific log file on indexer

timmy13 — Mon, 28 Sep 2020 11:26:38 GMT

MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRANSFORMS-Filter_Events = FilterNOVlog
TRANSFORMS-asa = cisco_asa
TRANSFORMS-fwsm = cisco_fwsm
TRANSFORMS-ios = cisco_ios
TRANSFORMS-pix = cisco_pix
TRUNCATE = 10000
maxDist = 100

Re: Filtering specific log file on indexer

lguinn2 — Mon, 27 Feb 2012 03:44:43 GMT

Try

[source::...\\NOV.log]

Re: Filtering specific log file on indexer

timmy13 — Mon, 27 Feb 2012 14:42:24 GMT

THanks, but sme result.

Re: Filtering specific log file on indexer

_d_ — Mon, 27 Feb 2012 15:01:25 GMT

Timmy13, are you saying that you tried this on the indexer and it does not work?

props.conf

[source::C:\\Program Files\\Common Files\\xxx\\xxx.xx.ProcessorService\\xxx\\logs\\NOV.log] TRANSFORMS-Filter_Events = FilterNOVlog

transforms.conf

[FilterNOVlog] REGEX = . DEST_KEY = queue FORMAT = nullQueue

Re: Filtering specific log file on indexer

timmy13 — Mon, 27 Feb 2012 15:12:42 GMT

Unfortunately, yes, tried that, and every other possible props.conf I can think of, none of them are working.

Re: Filtering specific log file on indexer

_d_ — Mon, 27 Feb 2012 15:29:01 GMT

hmm, this is very strange...my next step would be to investigate the indexer configuration with btool on props and transforms (with both, source and sourcetype stanzas) to make sure there are no conflicting settings...

Re: Filtering specific log file on indexer

timmy13 — Thu, 08 Mar 2012 18:26:38 GMT

Splunk support explains that this is failing because the data is being sent by the full forwarder, and therefore, the filter would have to be done on the forwarder.