topic Re: Blacklist question in Getting Data In

Blacklist question

ggouillart — Tue, 06 Aug 2019 13:56:39 GMT

Dear all,

I would like to blacklist the INFO logs from multiple sources. I have a log that looks like this:
Aug 6 15:52:12 SERVERNAMEvpxd[2568] 2019-08-06T15:52:12.394+02:00 info vpxd[7FE257707700] [Originator@6876 sub=vpxLro opID=69ff7094-a352-4198-8d64-745913c1a13c-3376170-ngc-89] [VpxLRO] -- BEGIN lro-1471645 -- ChangeLogCollector -- vim.cdc.ChangeLogCollector.waitForChanges -- 5274327a-9694-a5a6-e539-3c4d97209e25(52f849af-3efe-84e0-c365-6c2a1d5e0cec)

I want to blacklist every log that contains "INFO " or "info " (contains a space after info or INFO). My blacklist for each source is
blacklist2 = "INFO " (or blacklist5 etc).

So far it did not work. I've checked some of the questions and the doc and can't find the answer. Anyone has the answer?

By the way, this field is not parsed correctly. When I try to look for the logs without INFO I search index=nameoftheindex NOT "INFO "

Thanks in advance for your help.

Best regards,

Re: Blacklist question

gaurav_maniar — Wed, 30 Sep 2020 01:39:22 GMT

Hi,

You can set configuration in props.conf & transforms.conf with following to exclude "INFO" events from indexing.

props.conf

[source_type_name]
TRANSFORMS-null= setnull
transforms.conf

[setnull]
REGEX = +\d+:\d+\s(INFO|info)\s
DEST_KEY = queue
FORMAT = nullQueue

Accept and upvote the answer if it helps.

Re: Blacklist question

ggouillart — Wed, 07 Aug 2019 13:08:31 GMT

Should these settings be made on the universal forwarder or on the indexer? Would these logs considered as indexed? The main point is to lower the daily indexation volume.

Thanks for you answer

Re: Blacklist question

gaurav_maniar — Wed, 07 Aug 2019 13:59:42 GMT

It requires Heavy forwarder.
But if you are not using Heavy Forwarder, you can put these settings on Indexers.

The discarded data won't be counted against your license volume.