topic Re: Indexers not processing "Discard specific events and keep the rest" in Getting Data In

Why are Indexers not processing "Discard specific events and keep the rest"?

_olivier_ — Tue, 20 Jun 2023 05:30:40 GMT

Hi,

I wana keep only logs Not containing the word "chatbot".

This word is present in the _raw data

I'm using the method explained in the following doc : Routeandfilterdatad

The props.conf and transforms.conf are set on the indexers and I restarts my indexers

But logs with this word are still present.

Any idea, or way to debug this point ?

props.conf

[MySourcetype] INDEXED_EXTRACTIONS = JSON TIME_PREFIX=\"timestamp\": TIME_FORMAT=%s%3N #Do not index chatbot data TRANSFORMS-null = API-NullQueue

transforms.conf

[API-NullQueue] REGEX = chatbot DEST_KEY = queue FORMAT = nullQueue

Thank's all.

Re: Indexers not processing "Discard specific events and keep the rest"

gcusello — Fri, 16 Jun 2023 10:00:42 GMT

Hi @_olivier_,

this configuration must be located on Indexers or (when present) on intemediate Heavy Forwarders.

have you intermediate HFs in your architecture?

Did you checked the regex you are using? in other words in eachevent to discard is the "chatbot" word present?

Ciao.

Giuseppe

Re: Indexers not processing "Discard specific events and keep the rest"

_olivier_ — Fri, 16 Jun 2023 10:08:31 GMT

Hi,

Thank you for your rapid answer !

I have no HF on this part of my network, only UF forwarding data to indexers

I checked the regex on regex101 this word is matching each line I need to send to nullqueue.

Re: Indexers not processing "Discard specific events and keep the rest"

gcusello — Fri, 16 Jun 2023 10:47:29 GMT

Hi @_olivier_,

let me understand:

you ingest these logs in one or more Universal Forwarders,
these UFs directly send their logs to Indexers without any intermediate HF,
the sourcetype you assign on UF is the same that you used in the props.conf on Indexers,

only two final (very stupid) questions:

did you inserted the above conf files in all your Indexers?
did you restarted Splunk on Indexers after conf files update?

Ciao.

Giuseppe

Re: Indexers not processing "Discard specific events and keep the rest"

PickleRick — Fri, 16 Jun 2023 10:58:54 GMT

I seem to recall something about index-time operation not working when used with indexed extractions.

Also, if you're using 9.0 you can use Ingest Actions to filter data.

Re: Indexers not processing "Discard specific events and keep the rest"

_olivier_ — Fri, 16 Jun 2023 12:25:31 GMT

@gcusello , yes to all the final (not stupid) questions !

@PickleRick , My servers are 8.2.5, maybe a point to upgrade !

I will open a case and come back with their advises.

Thank's all!

Olivier

Re: Indexers not processing "Discard specific events and keep the rest"

gcusello — Fri, 16 Jun 2023 13:05:58 GMT

Hi @_olivier_,

ok, the only dubt is the one mentioned by @PickleRick: I searched in documentatio by I didn't find any information about this.

So, to be more sure: open a case to Splunk Support, they will surely and quicly give you the correct answer.

Ciao.

Giuseppe

Re: Indexers not processing "Discard specific events and keep the rest"

PickleRick — Sat, 17 Jun 2023 09:32:51 GMT

There doesn't seem to be a direct mention about that in docs.

But it does make sense. If you set indexed_extractions, the extraction is done _at the UF level_ when the file is read. So it is pushed further downstream in parsed form, not cooked. So subsequent components do not run props/transforms.