Why are Indexers not processing "Discard specific events and keep the rest"?

[_olivier_]

Hi,

I wana keep only logs Not containing the word "chatbot".

This word is present in the _raw data

I'm using the method explained in the following doc : Routeandfilterdatad 

The props.conf and transforms.conf are set on the indexers and I restarts my indexers

But logs with this word are still present.

Any idea, or way to debug this point ?

props.conf

[MySourcetype]
INDEXED_EXTRACTIONS = JSON
TIME_PREFIX=\"timestamp\":
TIME_FORMAT=%s%3N

#Do not index chatbot data
TRANSFORMS-null = API-NullQueue

transforms.conf

[API-NullQueue]
REGEX = chatbot
DEST_KEY = queue
FORMAT = nullQueue

Thank's all.

[PickleRick]

I seem to recall something about index-time operation not working when used with indexed extractions.

Also, if you're using 9.0 you can use Ingest Actions to filter data.

[gcusello]

Hi @_olivier_,

this configuration must be located on Indexers or (when present) on intemediate Heavy Forwarders.

have you intermediate HFs in your architecture?

Did you checked the regex you are using? in other words in eachevent to discard is the "chatbot" word present?

Ciao.

Giuseppe

[_olivier_]

Hi, 

Thank you for your rapid answer !

I have no HF on this part of my network, only UF forwarding data to indexers

I checked the regex on regex101 this word is matching each line I need to send to nullqueue.  

[gcusello]

Hi @_olivier_,

let me understand:

• you ingest these logs in one or more Universal Forwarders,
• these UFs directly send their logs to Indexers without any intermediate HF,
• the sourcetype you assign on UF is the same that you used in the props.conf on Indexers,

only two final (very stupid) questions:

• did you inserted the above conf files in all your Indexers?
• did you restarted Splunk on Indexers after conf files update?

Ciao.

Giuseppe

[_olivier_]

@gcusello , yes to all the final (not stupid) questions !

@PickleRick , My servers are 8.2.5, maybe a point to upgrade !

I will open a case and come back with their advises.

Thank's all!

Olivier

[gcusello]

Hi @_olivier_,

ok, the only dubt is the one mentioned by @PickleRick: I searched in documentatio by I didn't find any information about this.

So, to be more sure: open a case to Splunk Support, they will surely and quicly give you the correct answer.

Ciao.

Giuseppe

[PickleRick]

There doesn't seem to be a direct mention about that in docs.

But it does make sense. If you set indexed_extractions, the extraction is done _at the UF level_ when the file is read. So it is pushed further downstream in parsed form, not cooked. So subsequent components do not run props/transforms.