Filtering specific log file on indexer

[timmy13]

I have a very chatty forwarder that I do not have access to, so cannot stop the noise.

I have identified the log file that is sending millions of events an hour.

I would like to use props.conf and transforms.conf to filter this log out pre-index time, but it's not working like I would expect. The log file (NOV.log) is in a deep path, but I just want to use the name and no worry about the full path.

Here's my config:

props.conf
[source::...NOV.log}
TRANSFORMS-Filter_Events = FilterNOVlog

transforms.conf
[FilterNOVlog]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

Am I missing something really stupid, or shouldn't this work?

Thanks in advance!

Tim

[_d_]

Timmy13, are you saying that you tried this on the indexer and it does not work?

props.conf

[source::C:\\Program Files\\Common Files\\xxx\\xxx.xx.ProcessorService\\xxx\\logs\\NOV.log]
TRANSFORMS-Filter_Events = FilterNOVlog

transforms.conf

[FilterNOVlog]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[timmy13]

Splunk support explains that this is failing because the data is being sent by the full forwarder, and therefore, the filter would have to be done on the forwarder.

[_d_]

hmm, this is very strange...my next step would be to investigate the indexer configuration with btool on props and transforms (with both, source and sourcetype stanzas) to make sure there are no conflicting settings...

[timmy13]

Unfortunately, yes, tried that, and every other possible props.conf I can think of, none of them are working.

[lguinn2]

Try

[source::...\\NOV.log]

[timmy13]

THanks, but sme result.

[_d_]

Your props and transforms look correctly set for nullQueue-ing. Here are a couple of things you can try to troubleshoot the issue:

1. Is the source of the that data actually ...NOV.log? Perhaps it is being overwritten at inputs.conf at the forwarder.
2. Use btool to see the running configuration of Splunk for that source: ./splunk cmd btool props list my_source

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

[timmy13]

MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRANSFORMS-Filter_Events = FilterNOVlog
TRANSFORMS-asa = cisco_asa
TRANSFORMS-fwsm = cisco_fwsm
TRANSFORMS-ios = cisco_ios
TRANSFORMS-pix = cisco_pix
TRUNCATE = 10000
maxDist = 100

[timmy13]

Yes, the source is actually C:\Program Files\Common Files\xxx\xxx.xx.ProcessorService\xxx\logs\NOV.log, but I tried it with the full path as well and it didn't work.

Here's the btool output...
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
LEARN_SOURCETYPE = true
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800

[jgedeon120]

Yes that should work, after you correct the } to a ]. Have you restarted Splunk?

[timmy13]

Yah, I noticed that typo too, but it isn't working. ANd yes, I restarted splunk.